[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:22 dist-test-slave-0kls krb5kdc[3010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:22 dist-test-slave-0kls krb5kdc[3010](info): set up 2 sockets
Aug 11 04:36:22 dist-test-slave-0kls krb5kdc[3010](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260811 04:36:23.660140 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.111s user 0.004s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:23 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:23.794368 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40913
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:40913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:23.911936 3025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:23.912235 3025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:23.912276 3025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:23.916338 3025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:23.916409 3025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:23.916427 3025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:23.916443 3025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:23.916457 3025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:23.921988 3025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:40913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:23.923105 3025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:23.924187 3025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:23.933715 3031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:23.933815 3025 server_base.cc:1034] running on GCE node
W20260811 04:36:23.934733 3032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:23.934962 3034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:23.935622 3025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:23.936911 3025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:23.938107 3025 hybrid_clock.cc:648] HybridClock initialized: now 1786422983938100 us; error 64 us; skew 500 ppm
Aug 11 04:36:23 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422983, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:23.941061 3025 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:23.942344 3025 webserver.cc:460] Webserver started at http://127.2.238.190:36991/ using document root <none> and password file <none>
I20260811 04:36:23.942971 3025 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:23.943071 3025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:23.943308 3025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:23.945475 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2"
format_stamp: "Formatted at 2026-08-11 04:36:23 on dist-test-slave-0kls"
server_key: "6a002acc68d90e3b8e615542f2402723"
server_key_iv: "9fb4c48324e83855fee680e4a431a6e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:23.946076 3025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2"
format_stamp: "Formatted at 2026-08-11 04:36:23 on dist-test-slave-0kls"
server_key: "6a002acc68d90e3b8e615542f2402723"
server_key_iv: "9fb4c48324e83855fee680e4a431a6e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:23.950186 3025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:36:23.952808 3041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:23.954247 3025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:23.954398 3025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2"
format_stamp: "Formatted at 2026-08-11 04:36:23 on dist-test-slave-0kls"
server_key: "6a002acc68d90e3b8e615542f2402723"
server_key_iv: "9fb4c48324e83855fee680e4a431a6e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:23.954609 3025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:23.972469 3025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:23.975919 3025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:23.976143 3025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:23.986815 3025 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:40913
I20260811 04:36:23.986904 3096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:40913 every 8 connection(s)
I20260811 04:36:23.987988 3025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:23.991044 3097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:23.993006 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3025
I20260811 04:36:23.993201 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:23.995330 3002 external_mini_cluster.cc:1428] Setting key 402a00e642f32411a44b7f68d86a0d09
I20260811 04:36:24.000197 3097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2: Bootstrap starting.
I20260811 04:36:24.004739 3097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:24.006718 3097 log.cc:824] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:24.009989 3097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2: No bootstrap required, opened a new log
I20260811 04:36:24.014825 3097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } }
I20260811 04:36:24.015057 3097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:24.015102 3097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953d5ca3112e43c3b0ba8a9b8d36b2c2, State: Initialized, Role: FOLLOWER
I20260811 04:36:24.015694 3097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } }
I20260811 04:36:24.015865 3097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:24.015959 3097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:24.016078 3097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:24.017181 3097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } }
I20260811 04:36:24.017553 3097 leader_election.cc:302] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953d5ca3112e43c3b0ba8a9b8d36b2c2; no voters:
I20260811 04:36:24.017995 3097 leader_election.cc:288] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:24.018165 3102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:24.018415 3102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [term 1 LEADER]: Becoming Leader. State: Replica: 953d5ca3112e43c3b0ba8a9b8d36b2c2, State: Running, Role: LEADER
I20260811 04:36:24.018884 3102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } }
I20260811 04:36:24.019531 3097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:24.020287 3102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953d5ca3112e43c3b0ba8a9b8d36b2c2. Latest consensus state: current_term: 1 leader_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } } }
I20260811 04:36:24.020694 3102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [sys.catalog]: This master's current role is: LEADER
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:24.022035 3103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953d5ca3112e43c3b0ba8a9b8d36b2c2" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40913 } } }
I20260811 04:36:24.022166 3103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:24.023661 3108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:24.027642 3108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:24.031635 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:23.999438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.2.238.190:40913)
0811 04:36:24.000314 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.000380 (+ 66us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.001337 (+ 957us) server_negotiation.cc:407] Connection header received
0811 04:36:24.008250 (+ 6913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.008277 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.008728 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.009095 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.012676 (+ 3581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.014074 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.015730 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.016479 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.016714 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.023359 (+ 6645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.023370 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.023388 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.023431 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.026507 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.027567 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.027573 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.027579 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.027789 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.028258 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.028261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.028263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.028912 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.029215 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.029988 (+ 773us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.030362 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":351,"threads_started":1}
I20260811 04:36:24.035046 3108 catalog_manager.cc:1347] Generated new cluster ID: 1e7cee3ef81b49238ef8c383d870d6f0
I20260811 04:36:24.035137 3108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:24.056485 3108 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:24.057678 3108 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:24.064236 3108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 953d5ca3112e43c3b0ba8a9b8d36b2c2: Generated new TSK 0
I20260811 04:36:24.064972 3108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:24.101203 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:24.248128 3125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:24.248421 3125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:24.248584 3125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:24.252904 3125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:24.252975 3125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:24.253033 3125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:24.258767 3125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:24.260063 3125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:24.261221 3125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:24.271698 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:24.271782 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:24.271814 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:24.271971 3125 server_base.cc:1034] running on GCE node
I20260811 04:36:24.272550 3125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:24.273355 3125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:24.274518 3125 hybrid_clock.cc:648] HybridClock initialized: now 1786422984274475 us; error 55 us; skew 500 ppm
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:24.277343 3125 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:24.278502 3125 webserver.cc:460] Webserver started at http://127.2.238.129:45705/ using document root <none> and password file <none>
I20260811 04:36:24.279120 3125 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:24.279184 3125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:24.279455 3125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:24.281684 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "6cafcab34c224044a44cfdce019294f0"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "5817f4bcfbd3ac59ea3cea0caface9d9"
server_key_iv: "ef06d2b02341c20363a7755c9fde5d27"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.282344 3125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "6cafcab34c224044a44cfdce019294f0"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "5817f4bcfbd3ac59ea3cea0caface9d9"
server_key_iv: "ef06d2b02341c20363a7755c9fde5d27"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.286614 3125 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:24.289323 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.290797 3125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:36:24.290937 3125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "6cafcab34c224044a44cfdce019294f0"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "5817f4bcfbd3ac59ea3cea0caface9d9"
server_key_iv: "ef06d2b02341c20363a7755c9fde5d27"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.291046 3125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:24.304018 3125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:24.306936 3125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:24.307149 3125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:24.307785 3125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:24.308670 3125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:24.308743 3125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.308811 3125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:24.308854 3125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.324837 3125 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:39325
I20260811 04:36:24.324857 3263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:39325 every 8 connection(s)
I20260811 04:36:24.325860 3125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:24.329818 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3125
I20260811 04:36:24.329953 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:24.330229 3002 external_mini_cluster.cc:1428] Setting key 723dde96d1f98673c016c0268586c3f3
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:24.341946 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.327937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:58905 (local address 127.2.238.190:40913)
0811 04:36:24.328112 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.328117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.329014 (+ 897us) server_negotiation.cc:407] Connection header received
0811 04:36:24.330812 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.330815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.330874 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.331019 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.332443 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.332954 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.334753 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.334986 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.335105 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.337751 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.337754 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.337756 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.337835 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.339327 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.340014 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.340016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.340017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.340094 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.340547 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.340550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.340551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.340743 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.340830 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.341607 (+ 777us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.341751 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 04:36:24.342887 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.328369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40913 (local address 127.2.238.129:58905)
0811 04:36:24.328819 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:24.328865 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:24.330577 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:24.331205 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:24.331213 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:24.331577 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:24.332275 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.332302 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.333122 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.333129 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.334593 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.334605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.335157 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.335161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.335341 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.336008 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:24.336026 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:24.337521 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:24.339457 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.339464 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.339484 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.339911 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.340195 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.340199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.340200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.340411 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.340881 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:24.340893 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:24.341259 (+ 366us) client_negotiation.cc:769] Sending connection context
0811 04:36:24.341530 (+ 271us) client_negotiation.cc:240] Negotiation successful
0811 04:36:24.341949 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:36:24.344316 3264 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40913
I20260811 04:36:24.344609 3264 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:24.345140 3264 heartbeater.cc:511] Master 127.2.238.190:40913 requested a full tablet report, sending...
I20260811 04:36:24.346815 3059 ts_manager.cc:194] Registered new tserver with Master: 6cafcab34c224044a44cfdce019294f0 (127.2.238.129:39325)
I20260811 04:36:24.348567 3059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:58905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:24.360410 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:24.476044 3272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:24.476303 3272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:24.476359 3272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:24.480324 3272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:24.480378 3272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:24.480437 3272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:24.485423 3272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:24.486426 3272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:24.487440 3272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:24.497507 3280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:24.497510 3277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:24.497766 3272 server_base.cc:1034] running on GCE node
W20260811 04:36:24.497629 3278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:24.498287 3272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:24.498908 3272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:24.500069 3272 hybrid_clock.cc:648] HybridClock initialized: now 1786422984500032 us; error 49 us; skew 500 ppm
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:24.502851 3272 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:24.503917 3272 webserver.cc:460] Webserver started at http://127.2.238.130:45363/ using document root <none> and password file <none>
I20260811 04:36:24.504498 3272 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:24.504560 3272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:24.504829 3272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:24.506954 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "0a6e0a3c65b04c76b02da979cd89ed0c"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "0b5ffba34521dd5623098130c02b011a"
server_key_iv: "d6535113e4b5b6c33d5d61a3dbe9d33c"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.507563 3272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "0a6e0a3c65b04c76b02da979cd89ed0c"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "0b5ffba34521dd5623098130c02b011a"
server_key_iv: "d6535113e4b5b6c33d5d61a3dbe9d33c"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.511615 3272 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:24.513985 3287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.515259 3272 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:24.515410 3272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "0a6e0a3c65b04c76b02da979cd89ed0c"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "0b5ffba34521dd5623098130c02b011a"
server_key_iv: "d6535113e4b5b6c33d5d61a3dbe9d33c"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.515527 3272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:24.539326 3272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:24.542441 3272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:24.542654 3272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:24.543416 3272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:24.544278 3272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:24.544348 3272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.544421 3272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:24.544471 3272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.556982 3272 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43607
I20260811 04:36:24.557003 3410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43607 every 8 connection(s)
I20260811 04:36:24.558022 3272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:24.558396 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3272
I20260811 04:36:24.558547 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:24.558883 3002 external_mini_cluster.cc:1428] Setting key 2175d1896f0bf77c0923ab1aea012b30
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
I20260811 04:36:24.573582 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.560241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:38997 (local address 127.2.238.190:40913)
0811 04:36:24.560413 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.560418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.561514 (+ 1096us) server_negotiation.cc:407] Connection header received
0811 04:36:24.562909 (+ 1395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.562912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.562965 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.563043 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.564325 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.564873 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.566263 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.566499 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.566611 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.569473 (+ 2862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.569476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.569477 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.569507 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.571116 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.571576 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.571578 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.571579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.571652 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.572028 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.572031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.572032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.572212 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.572303 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.573095 (+ 792us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.573254 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:36:24.574288 3414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.560763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40913 (local address 127.2.238.130:38997)
0811 04:36:24.561376 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:24.561418 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:24.562694 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:24.563212 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:24.563218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:24.563670 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:24.564179 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.564192 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.565009 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.565018 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.566083 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.566112 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.566666 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.566670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.566843 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.567448 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:24.567465 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:24.569290 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:24.571224 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.571229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.571243 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.571489 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.571739 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.571742 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.571743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.571917 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.572318 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:24.572322 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:24.572696 (+ 374us) client_negotiation.cc:769] Sending connection context
0811 04:36:24.573069 (+ 373us) client_negotiation.cc:240] Negotiation successful
0811 04:36:24.573354 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":265,"threads_started":1}
I20260811 04:36:24.575555 3411 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40913
I20260811 04:36:24.575857 3411 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:24.576471 3411 heartbeater.cc:511] Master 127.2.238.190:40913 requested a full tablet report, sending...
I20260811 04:36:24.577540 3059 ts_manager.cc:194] Registered new tserver with Master: 0a6e0a3c65b04c76b02da979cd89ed0c (127.2.238.130:43607)
I20260811 04:36:24.578182 3059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:38997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:24.588486 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:24.706470 3417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:24.706745 3417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:24.706791 3417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:24.710829 3417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:24.710889 3417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:24.710951 3417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:24.716199 3417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40913
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:24.717275 3417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:24.718401 3417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:24.728609 3422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:24.728731 3417 server_base.cc:1034] running on GCE node
W20260811 04:36:24.728609 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:24.728618 3423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:24.729513 3417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:24.730307 3417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:24.731475 3417 hybrid_clock.cc:648] HybridClock initialized: now 1786422984731453 us; error 33 us; skew 500 ppm
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:24.734629 3417 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:24.735584 3417 webserver.cc:460] Webserver started at http://127.2.238.131:38801/ using document root <none> and password file <none>
I20260811 04:36:24.736114 3417 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:24.736171 3417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:24.736572 3417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:24.738859 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "360856c59df04053a4dc3ba4206cc62e"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "1b7c98f39fe96511924434a0fdcc5a33"
server_key_iv: "d52ccf4e5c1c7fe8ac2c3f8e845cef03"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.739398 3417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "360856c59df04053a4dc3ba4206cc62e"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "1b7c98f39fe96511924434a0fdcc5a33"
server_key_iv: "d52ccf4e5c1c7fe8ac2c3f8e845cef03"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.743424 3417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:24.746291 3433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.747591 3417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:24.747694 3417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "360856c59df04053a4dc3ba4206cc62e"
format_stamp: "Formatted at 2026-08-11 04:36:24 on dist-test-slave-0kls"
server_key: "1b7c98f39fe96511924434a0fdcc5a33"
server_key_iv: "d52ccf4e5c1c7fe8ac2c3f8e845cef03"
server_key_version: "encryptionkey@0"
I20260811 04:36:24.747790 3417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:24.759038 3417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:24.762207 3417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:24.762382 3417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:24.762950 3417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:24.763804 3417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:24.763852 3417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.763893 3417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:24.763916 3417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:24.775606 3417 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:44233
I20260811 04:36:24.775628 3555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:44233 every 8 connection(s)
I20260811 04:36:24.776667 3417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:24.786381 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3417
I20260811 04:36:24.786536 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:24.786862 3002 external_mini_cluster.cc:1428] Setting key 3156b2d9b5c34f3bb86e1e8ad7e67019
I20260811 04:36:24.791218 3099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.778841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:42221 (local address 127.2.238.190:40913)
0811 04:36:24.779021 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.779026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.780000 (+ 974us) server_negotiation.cc:407] Connection header received
0811 04:36:24.781393 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.781396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.781445 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.781523 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.782768 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.783298 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.784406 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.784610 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.784686 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.787391 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.787400 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.787402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.787428 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.789036 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.789576 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.789580 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.789582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.789673 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.790076 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.790079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.790081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.790289 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.790402 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.790922 (+ 520us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.791051 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 04:36:24.792026 3558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.779346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40913 (local address 127.2.238.131:42221)
0811 04:36:24.779831 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:24.779876 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:24.781156 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:24.781691 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:24.781698 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:24.782134 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:24.782629 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.782642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.783403 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.783405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.784302 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:24.784311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.784748 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.784751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:24.784888 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.785435 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:24.785451 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:24.787190 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:24.789166 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.789172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.789188 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.789464 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.789779 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:24.789799 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:24.789803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:24.789964 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:24.790401 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:24.790408 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:24.790665 (+ 257us) client_negotiation.cc:769] Sending connection context
0811 04:36:24.790863 (+ 198us) client_negotiation.cc:240] Negotiation successful
0811 04:36:24.791198 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260811 04:36:24.793089 3556 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40913
I20260811 04:36:24.793365 3556 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:24.793929 3556 heartbeater.cc:511] Master 127.2.238.190:40913 requested a full tablet report, sending...
I20260811 04:36:24.795006 3059 ts_manager.cc:194] Registered new tserver with Master: 360856c59df04053a4dc3ba4206cc62e (127.2.238.131:44233)
I20260811 04:36:24.795603 3059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:42221
I20260811 04:36:24.801491 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:24.829676 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.820141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35162 (local address 127.2.238.129:39325)
0811 04:36:24.820525 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.820541 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.820566 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:36:24.820678 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.820684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.820880 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.821003 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.821623 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.822490 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.823371 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.823786 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.823891 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.826260 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.826270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.826284 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.826309 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.827985 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.828370 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.828374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.828376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.828455 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.828679 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.828682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.828683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.828933 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.829082 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.829247 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.829413 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":159,"threads_started":1}
W20260811 04:36:24.830695 3172 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35162
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:24.858785 3563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:24.849765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.2.238.129:39325)
0811 04:36:24.849947 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:24.849952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:24.850072 (+ 120us) server_negotiation.cc:407] Connection header received
0811 04:36:24.850238 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:24.850241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:24.850293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:24.850385 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:24.851091 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.851666 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.852692 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:24.852954 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:24.853088 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:24.855631 (+ 2543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:24.855635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:24.855637 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:24.855665 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:24.857016 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.857466 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.857470 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.857471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.857553 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:24.857880 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:24.857883 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:24.857885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:24.858070 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:24.858207 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:24.858488 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 04:36:24.858614 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 04:36:24.859999 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3125
I20260811 04:36:24.868994 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3272
I20260811 04:36:24.879583 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3417
I20260811 04:36:24.886942 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3025
2026-08-11T04:36:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2375 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3570](info): set up 2 sockets
Aug 11 04:36:24 dist-test-slave-0kls krb5kdc[3570](info): commencing operation
krb5kdc: starting...
W20260811 04:36:25.958469 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:26.093057 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:37293
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:37293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:26.207544 3584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:26.207898 3584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:26.207973 3584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:26.212101 3584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:26.212194 3584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:26.212230 3584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:26.212268 3584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:26.212301 3584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:26.217640 3584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:37293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:37293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:26.218851 3584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:26.219961 3584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:26.228955 3591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:26.229028 3593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:26.229161 3584 server_base.cc:1034] running on GCE node
W20260811 04:36:26.229028 3590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:26.230080 3584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:26.231217 3584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:26.232407 3584 hybrid_clock.cc:648] HybridClock initialized: now 1786422986232373 us; error 48 us; skew 500 ppm
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:26.235420 3584 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:26.236342 3584 webserver.cc:460] Webserver started at http://127.2.238.190:42831/ using document root <none> and password file <none>
I20260811 04:36:26.236917 3584 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:26.237007 3584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:26.237252 3584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:26.239451 3584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "6b74331ada9f40849514bf47fa7d4a6d"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "13010761466a4bb4c1745b4aa6f5100d"
server_key_iv: "d95c77340250c4a75b2845f6bac6b92f"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.240209 3584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "6b74331ada9f40849514bf47fa7d4a6d"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "13010761466a4bb4c1745b4aa6f5100d"
server_key_iv: "d95c77340250c4a75b2845f6bac6b92f"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.244261 3584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:26.246892 3604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.248140 3584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:26.248286 3584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "6b74331ada9f40849514bf47fa7d4a6d"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "13010761466a4bb4c1745b4aa6f5100d"
server_key_iv: "d95c77340250c4a75b2845f6bac6b92f"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.248409 3584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:26.262115 3584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:26.265400 3584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:26.265623 3584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:26.273300 3584 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:37293
I20260811 04:36:26.273315 3659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:37293 every 8 connection(s)
I20260811 04:36:26.274336 3584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:26.277194 3660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:26.280426 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3584
I20260811 04:36:26.280544 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:26.280817 3002 external_mini_cluster.cc:1428] Setting key 392b2d4b6c40619eeb5e71608cdf3a27
I20260811 04:36:26.283775 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d: Bootstrap starting.
I20260811 04:36:26.286103 3660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:26.286885 3660 log.cc:824] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:26.289288 3660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d: No bootstrap required, opened a new log
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:26.292507 3660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } }
I20260811 04:36:26.292770 3660 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:26.292851 3660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b74331ada9f40849514bf47fa7d4a6d, State: Initialized, Role: FOLLOWER
I20260811 04:36:26.293298 3660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } }
I20260811 04:36:26.293419 3660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:26.293517 3660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:26.293627 3660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:26.294898 3660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } }
I20260811 04:36:26.295271 3660 leader_election.cc:302] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b74331ada9f40849514bf47fa7d4a6d; no voters:
I20260811 04:36:26.295660 3660 leader_election.cc:288] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:26.295797 3665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:26.296021 3665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [term 1 LEADER]: Becoming Leader. State: Replica: 6b74331ada9f40849514bf47fa7d4a6d, State: Running, Role: LEADER
I20260811 04:36:26.296379 3665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } }
I20260811 04:36:26.296972 3660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:26.297114 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:26.282409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.2.238.190:37293)
0811 04:36:26.283041 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:26.283058 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:26.283116 (+ 58us) server_negotiation.cc:407] Connection header received
0811 04:36:26.283744 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:26.283760 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:26.284169 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:26.284519 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:26.285348 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.286490 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.287482 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.287955 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.288082 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:26.290924 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:26.290929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:26.290943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:26.290970 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:26.293766 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.294309 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.294314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.294319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.294430 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.294734 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.294736 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.294737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.295186 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:26.295330 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:26.295700 (+ 370us) server_negotiation.cc:299] Negotiation successful
0811 04:36:26.296110 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":155,"threads_started":1}
I20260811 04:36:26.298624 3666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b74331ada9f40849514bf47fa7d4a6d. Latest consensus state: current_term: 1 leader_uuid: "6b74331ada9f40849514bf47fa7d4a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } } }
I20260811 04:36:26.298766 3666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:26.299113 3668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b74331ada9f40849514bf47fa7d4a6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b74331ada9f40849514bf47fa7d4a6d" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37293 } } }
I20260811 04:36:26.299198 3668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:26.300347 3678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:26.303444 3678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:26.309190 3678 catalog_manager.cc:1347] Generated new cluster ID: 848371492ae1422b8d18e338d30d7c98
I20260811 04:36:26.309264 3678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:26.336936 3678 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:26.338153 3678 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:26.352628 3678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b74331ada9f40849514bf47fa7d4a6d: Generated new TSK 0
I20260811 04:36:26.353541 3678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:26.391178 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:26.510198 3689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:26.510504 3689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:26.510553 3689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:26.514721 3689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:26.514786 3689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:26.514847 3689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:26.520562 3689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:26.521785 3689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:26.522915 3689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:26.533382 3696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:26.533380 3695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:26.533469 3698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:26.533818 3689 server_base.cc:1034] running on GCE node
I20260811 04:36:26.534305 3689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:26.535002 3689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:26.536193 3689 hybrid_clock.cc:648] HybridClock initialized: now 1786422986536176 us; error 27 us; skew 500 ppm
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:26.539237 3689 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:26.540227 3689 webserver.cc:460] Webserver started at http://127.2.238.129:34801/ using document root <none> and password file <none>
I20260811 04:36:26.540828 3689 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:26.540923 3689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:26.541177 3689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:26.543460 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "1212fb30ab954976b55aeea7cfabf29b"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "273a84f3c91b12a474ce4b5edd0b75c0"
server_key_iv: "1fc62f27618e0e9b468cba5312f0a5b9"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.544039 3689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "1212fb30ab954976b55aeea7cfabf29b"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "273a84f3c91b12a474ce4b5edd0b75c0"
server_key_iv: "1fc62f27618e0e9b468cba5312f0a5b9"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.547974 3689 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:26.550614 3707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.551885 3689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:36:26.552009 3689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "1212fb30ab954976b55aeea7cfabf29b"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "273a84f3c91b12a474ce4b5edd0b75c0"
server_key_iv: "1fc62f27618e0e9b468cba5312f0a5b9"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.552137 3689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:26.569383 3689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:26.572552 3689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:26.572779 3689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:26.573498 3689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:26.574463 3689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:26.574513 3689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.574584 3689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:26.574615 3689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.584118 3689 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:41857
I20260811 04:36:26.584139 3830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:41857 every 8 connection(s)
I20260811 04:36:26.585120 3689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:26.589015 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3689
I20260811 04:36:26.589164 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:26.589459 3002 external_mini_cluster.cc:1428] Setting key 0d10aed9e331388e5ee46174f7215fea
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:26.601336 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:26.587208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56075 (local address 127.2.238.190:37293)
0811 04:36:26.587415 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:26.587420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:26.588289 (+ 869us) server_negotiation.cc:407] Connection header received
0811 04:36:26.590098 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:26.590102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:26.590165 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:26.590241 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:26.591811 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.592333 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.593817 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.594220 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.594334 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:26.597319 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:26.597322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:26.597324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:26.597351 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:26.599046 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.599633 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.599637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.599639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.599730 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.600106 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.600111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.600113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.600356 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:26.600453 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:26.601017 (+ 564us) server_negotiation.cc:299] Negotiation successful
0811 04:36:26.601165 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260811 04:36:26.602288 3833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:26.587608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:37293 (local address 127.2.238.129:56075)
0811 04:36:26.588098 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:26.588165 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:26.589796 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:26.590450 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:26.590457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:26.590905 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:26.591641 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:26.591665 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.592469 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.592475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:26.593657 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:26.593669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.594390 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.594393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:26.594538 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:26.595272 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:26.595290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:26.597098 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:26.599208 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:26.599214 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:26.599235 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:26.599513 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:26.599838 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:26.599841 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:26.599843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:26.600005 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:26.600455 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:26.600465 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:26.600757 (+ 292us) client_negotiation.cc:769] Sending connection context
0811 04:36:26.600965 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 04:36:26.601345 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":127,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:36:26.603778 3831 heartbeater.cc:347] Connected to a master server at 127.2.238.190:37293
I20260811 04:36:26.604072 3831 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:26.604621 3831 heartbeater.cc:511] Master 127.2.238.190:37293 requested a full tablet report, sending...
I20260811 04:36:26.606377 3622 ts_manager.cc:194] Registered new tserver with Master: 1212fb30ab954976b55aeea7cfabf29b (127.2.238.129:41857)
I20260811 04:36:26.607545 3622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:56075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:26.620721 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:26.743137 3837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:26.743389 3837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:26.743463 3837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:26.747532 3837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:26.747593 3837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:26.747689 3837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:26.753100 3837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:26.754254 3837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:26.755337 3837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:26.765486 3842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:26.765506 3846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:26.765537 3837 server_base.cc:1034] running on GCE node
W20260811 04:36:26.765470 3843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:26.766180 3837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:26.766860 3837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:26.768033 3837 hybrid_clock.cc:648] HybridClock initialized: now 1786422986768028 us; error 38 us; skew 500 ppm
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:26.774642 3837 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:26.775614 3837 webserver.cc:460] Webserver started at http://127.2.238.130:36259/ using document root <none> and password file <none>
I20260811 04:36:26.776249 3837 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:26.776347 3837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:26.776597 3837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:26.778856 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "dff3ee9fe58e4696adba9d9d95b6f32a"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "79c9d4acebbb4df846e9ee2d7f8eb3e3"
server_key_iv: "8d587b60f6cbf1bd6dc39023faa29ccd"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.779430 3837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "dff3ee9fe58e4696adba9d9d95b6f32a"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "79c9d4acebbb4df846e9ee2d7f8eb3e3"
server_key_iv: "8d587b60f6cbf1bd6dc39023faa29ccd"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.783463 3837 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260811 04:36:26.785928 3854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.787202 3837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:26.787369 3837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "dff3ee9fe58e4696adba9d9d95b6f32a"
format_stamp: "Formatted at 2026-08-11 04:36:26 on dist-test-slave-0kls"
server_key: "79c9d4acebbb4df846e9ee2d7f8eb3e3"
server_key_iv: "8d587b60f6cbf1bd6dc39023faa29ccd"
server_key_version: "encryptionkey@0"
I20260811 04:36:26.787482 3837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:26.814488 3837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:26.818050 3837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:26.818275 3837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:26.818935 3837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:26.819918 3837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:26.819991 3837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.820066 3837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:26.820116 3837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:26.829253 3837 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:34527
I20260811 04:36:26.829280 3977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:34527 every 8 connection(s)
I20260811 04:36:26.830268 3837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:26.838995 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3837
I20260811 04:36:26.839143 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:26.839433 3002 external_mini_cluster.cc:1428] Setting key 53e3fe86c19167d26cc3c40755a499c9
Aug 11 04:36:26 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:26.844950 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:26.832197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41667 (local address 127.2.238.190:37293)
0811 04:36:26.832336 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:26.832340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:26.833136 (+ 796us) server_negotiation.cc:407] Connection header received
0811 04:36:26.834604 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:26.834607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:26.834684 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:26.834775 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:26.835975 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.836599 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.837767 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.838047 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.838166 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:26.840842 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:26.840846 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:26.840849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:26.840877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:26.842667 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.843335 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.843338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.843339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.843415 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:26.843821 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:26.843823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:26.843824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:26.843990 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:26.844064 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:26.844727 (+ 663us) server_negotiation.cc:299] Negotiation successful
0811 04:36:26.844830 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:26.846290 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:26.832536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:37293 (local address 127.2.238.130:41667)
0811 04:36:26.832982 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:26.833026 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:26.834378 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:26.834959 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:26.834966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:26.835336 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:26.835827 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:26.835841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.836748 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.836751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:26.837637 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:26.837647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:26.838211 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:26.838215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:26.838387 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:26.839013 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:26.839030 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:26.840652 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:26.842801 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:26.842806 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:26.842818 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:26.843222 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:26.843511 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:26.843517 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:26.843520 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:26.843726 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:26.844094 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:26.844101 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:26.844468 (+ 367us) client_negotiation.cc:769] Sending connection context
0811 04:36:26.844691 (+ 223us) client_negotiation.cc:240] Negotiation successful
0811 04:36:26.845061 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260811 04:36:26.847618 3978 heartbeater.cc:347] Connected to a master server at 127.2.238.190:37293
I20260811 04:36:26.848069 3978 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:26.848649 3978 heartbeater.cc:511] Master 127.2.238.190:37293 requested a full tablet report, sending...
I20260811 04:36:26.849872 3622 ts_manager.cc:194] Registered new tserver with Master: dff3ee9fe58e4696adba9d9d95b6f32a (127.2.238.130:34527)
I20260811 04:36:26.850490 3622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:41667
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:26.869145 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:26.999681 3984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:26.999982 3984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:27.000068 3984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:27.004297 3984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:27.004365 3984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:27.004426 3984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:27.010015 3984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:37293
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:27.011197 3984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:27.012264 3984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:27.022385 3990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:27.022385 3989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:27.022415 3992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:27.022714 3984 server_base.cc:1034] running on GCE node
I20260811 04:36:27.023200 3984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:27.023875 3984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:27.025038 3984 hybrid_clock.cc:648] HybridClock initialized: now 1786422987025024 us; error 25 us; skew 500 ppm
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:27.027967 3984 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:27.028855 3984 webserver.cc:460] Webserver started at http://127.2.238.131:37211/ using document root <none> and password file <none>
I20260811 04:36:27.029441 3984 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:27.029534 3984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:27.029815 3984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:27.031980 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "ce2b17d943774fecb830f612fb58025b"
format_stamp: "Formatted at 2026-08-11 04:36:27 on dist-test-slave-0kls"
server_key: "16448736c39bd4f09ba8e4c2557d3cb3"
server_key_iv: "abc5c21a95f817cdd51d477903b0137d"
server_key_version: "encryptionkey@0"
I20260811 04:36:27.032645 3984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "ce2b17d943774fecb830f612fb58025b"
format_stamp: "Formatted at 2026-08-11 04:36:27 on dist-test-slave-0kls"
server_key: "16448736c39bd4f09ba8e4c2557d3cb3"
server_key_iv: "abc5c21a95f817cdd51d477903b0137d"
server_key_version: "encryptionkey@0"
I20260811 04:36:27.036581 3984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 04:36:27.039148 4002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:27.040417 3984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:36:27.040557 3984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "ce2b17d943774fecb830f612fb58025b"
format_stamp: "Formatted at 2026-08-11 04:36:27 on dist-test-slave-0kls"
server_key: "16448736c39bd4f09ba8e4c2557d3cb3"
server_key_iv: "abc5c21a95f817cdd51d477903b0137d"
server_key_version: "encryptionkey@0"
I20260811 04:36:27.040683 3984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:27.050242 3984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:27.053272 3984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:27.053483 3984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:27.054101 3984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:27.054872 3984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:27.054917 3984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:27.054983 3984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:27.055023 3984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:27.063977 3984 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:45175
I20260811 04:36:27.064035 4127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:45175 every 8 connection(s)
I20260811 04:36:27.064953 3984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:27.066965 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 3984
I20260811 04:36:27.067104 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:27.067414 3002 external_mini_cluster.cc:1428] Setting key 3c6ead1ce9b1fedab182cee87f571699
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:27.078547 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.066960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:41273 (local address 127.2.238.190:37293)
0811 04:36:27.067155 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.067184 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.068137 (+ 953us) server_negotiation.cc:407] Connection header received
0811 04:36:27.069913 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.069916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.069962 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.070029 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:27.071142 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.071651 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.072706 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.072905 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.072975 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.075403 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:27.075406 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:27.075408 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:27.075431 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:27.076639 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.077094 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.077097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.077098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.077174 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.077459 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.077461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.077462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.077629 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:27.077711 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.078323 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.078424 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:36:27.079356 4130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.067393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:37293 (local address 127.2.238.131:41273)
0811 04:36:27.067961 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.068021 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.069668 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.070168 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.070175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.070532 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:27.071019 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.071033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.071754 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.071757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.072605 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.072614 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.073030 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.073032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.073183 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.073694 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:27.073709 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:27.075252 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:27.076757 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:27.076762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:27.076773 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:27.077011 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:27.077255 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:27.077257 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:27.077259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:27.077383 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:27.077722 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:27.077726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:27.078109 (+ 383us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.078273 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.078517 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":135,"threads_started":1}
I20260811 04:36:27.080574 4128 heartbeater.cc:347] Connected to a master server at 127.2.238.190:37293
I20260811 04:36:27.080869 4128 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:27.081490 4128 heartbeater.cc:511] Master 127.2.238.190:37293 requested a full tablet report, sending...
I20260811 04:36:27.082577 3622 ts_manager.cc:194] Registered new tserver with Master: ce2b17d943774fecb830f612fb58025b (127.2.238.131:45175)
I20260811 04:36:27.083128 3622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:41273
I20260811 04:36:27.093320 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:36:27.104463 3663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.096960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.2.238.190:37293)
0811 04:36:27.097107 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.097114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.097250 (+ 136us) server_negotiation.cc:407] Connection header received
0811 04:36:27.097345 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.097347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.097386 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.097449 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:27.098389 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.098945 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.100250 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.100460 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.100542 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.101841 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:27.101844 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:27.101846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:27.101867 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:27.103007 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.103389 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.103391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.103393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.103462 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.103680 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.103682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.103683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.103883 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:27.103982 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.104176 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.104271 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:36:27.109357 3622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:27.112309 3622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:27.127007 4140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.121025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45175 (local address 127.0.0.1:46304)
0811 04:36:27.121956 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.121970 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.122046 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.122886 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.122889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.122906 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:27.122985 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.122990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.124345 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.124348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.125395 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.125404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126493 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.126498 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.126652 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.126697 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.126788 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.126876 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":38,"thread_start_us":114,"threads_started":1}
I20260811 04:36:27.127053 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.121523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46304 (local address 127.2.238.131:45175)
0811 04:36:27.122415 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.122432 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.122453 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:36:27.122525 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.122530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.122740 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.122891 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:27.123136 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.124177 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.125523 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126346 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.126616 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.126772 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.126826 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.126911 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":155,"threads_started":1}
I20260811 04:36:27.127044 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.120863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:41857 (local address 127.0.0.1:44772)
0811 04:36:27.121385 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.121433 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.121554 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.122389 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.122395 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.122421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:27.122586 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.122592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.124046 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.124050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.125604 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.125615 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126648 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.126652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.126743 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.126766 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.126819 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.126889 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":76,"thread_start_us":74,"threads_started":1}
I20260811 04:36:27.127641 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.121551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:34527 (local address 127.0.0.1:54006)
0811 04:36:27.122482 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.122495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.122588 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.123081 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.123084 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.123101 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:27.123193 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.123198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.124191 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.124196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.125636 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.125646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126472 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.126476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.127351 (+ 875us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.127377 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.127461 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.127521 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":109,"thread_start_us":105,"threads_started":1}
I20260811 04:36:27.128100 4146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.121928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.2.238.130:34527)
0811 04:36:27.122360 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.122371 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.122592 (+ 221us) server_negotiation.cc:407] Connection header received
0811 04:36:27.122703 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.122710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.122882 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.122998 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:27.123305 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.124046 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.125777 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126346 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.126470 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.126630 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.127819 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.127929 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":116,"threads_started":1}
I20260811 04:36:27.128389 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.121031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44772 (local address 127.2.238.129:41857)
0811 04:36:27.121809 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.121817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.121843 (+ 26us) server_negotiation.cc:407] Connection header received
0811 04:36:27.121926 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.121932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.122144 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.122394 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:27.122722 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.123905 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.125739 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.126526 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.127460 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.127598 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.127645 (+ 47us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.128242 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":110,"threads_started":1}
I20260811 04:36:27.130733 3906 tablet_service.cc:1467] Processing CreateTablet for tablet 1d433b6451894b738b725d3b2c719607 (DEFAULT_TABLE table=test-table [id=3815d25f6ed94b75b9a3f7d90509d0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:27.130581 4055 tablet_service.cc:1467] Processing CreateTablet for tablet 1d433b6451894b738b725d3b2c719607 (DEFAULT_TABLE table=test-table [id=3815d25f6ed94b75b9a3f7d90509d0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:27.130963 3761 tablet_service.cc:1467] Processing CreateTablet for tablet 1d433b6451894b738b725d3b2c719607 (DEFAULT_TABLE table=test-table [id=3815d25f6ed94b75b9a3f7d90509d0da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:27.131765 3906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d433b6451894b738b725d3b2c719607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:27.131806 4055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d433b6451894b738b725d3b2c719607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:27.131851 3761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d433b6451894b738b725d3b2c719607. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:27.138986 4147 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: Bootstrap starting.
I20260811 04:36:27.139914 4151 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: Bootstrap starting.
I20260811 04:36:27.140472 4149 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: Bootstrap starting.
I20260811 04:36:27.142149 4147 tablet_bootstrap.cc:654] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:27.142781 4149 tablet_bootstrap.cc:654] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:27.143020 4151 tablet_bootstrap.cc:654] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:27.143121 4147 log.cc:824] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:27.143778 4149 log.cc:824] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:27.144023 4151 log.cc:824] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:27.145467 4147 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: No bootstrap required, opened a new log
I20260811 04:36:27.145679 4147 ts_tablet_manager.cc:1397] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260811 04:36:27.145946 4149 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: No bootstrap required, opened a new log
I20260811 04:36:27.145946 4151 tablet_bootstrap.cc:492] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: No bootstrap required, opened a new log
I20260811 04:36:27.146131 4149 ts_tablet_manager.cc:1397] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:36:27.146131 4151 ts_tablet_manager.cc:1397] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:36:27.148780 4147 raft_consensus.cc:348] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.149040 4147 raft_consensus.cc:374] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:27.149106 4147 raft_consensus.cc:729] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff3ee9fe58e4696adba9d9d95b6f32a, State: Initialized, Role: FOLLOWER
I20260811 04:36:27.149621 4147 consensus_queue.cc:260] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.150750 3978 heartbeater.cc:503] Master 127.2.238.190:37293 was elected leader, sending a full tablet report...
I20260811 04:36:27.150849 4147 ts_tablet_manager.cc:1428] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:36:27.151113 4151 raft_consensus.cc:348] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.151396 4151 raft_consensus.cc:374] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:27.151451 4151 raft_consensus.cc:729] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce2b17d943774fecb830f612fb58025b, State: Initialized, Role: FOLLOWER
I20260811 04:36:27.151410 4149 raft_consensus.cc:348] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.151693 4149 raft_consensus.cc:374] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:27.151754 4149 raft_consensus.cc:729] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1212fb30ab954976b55aeea7cfabf29b, State: Initialized, Role: FOLLOWER
I20260811 04:36:27.151903 4151 consensus_queue.cc:260] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.152141 4149 consensus_queue.cc:260] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.152788 3831 heartbeater.cc:503] Master 127.2.238.190:37293 was elected leader, sending a full tablet report...
I20260811 04:36:27.152894 4151 ts_tablet_manager.cc:1428] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20260811 04:36:27.152995 4149 ts_tablet_manager.cc:1428] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260811 04:36:27.153177 4128 heartbeater.cc:503] Master 127.2.238.190:37293 was elected leader, sending a full tablet report...
I20260811 04:36:27.163875 4156 raft_consensus.cc:482] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:27.164130 4156 raft_consensus.cc:504] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.165834 4156 leader_election.cc:288] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1212fb30ab954976b55aeea7cfabf29b (127.2.238.129:41857), ce2b17d943774fecb830f612fb58025b (127.2.238.131:45175)
I20260811 04:36:27.170976 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.166629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:42765 (local address 127.2.238.129:41857)
0811 04:36:27.166780 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.166786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.166806 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:36:27.167011 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.167015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.167124 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.167191 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:27.167605 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.168381 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.169691 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.170353 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.170465 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.170519 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.170787 (+ 268us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.170861 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:36:27.170989 4159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.166127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:41857 (local address 127.2.238.130:42765)
0811 04:36:27.166634 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.166650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.166836 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.167243 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.167247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.167264 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:27.167463 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.167469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.168514 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.168518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.169560 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.169570 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.170572 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.170574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.170646 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.170664 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.170770 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.170840 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":1}
I20260811 04:36:27.171742 4145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.166547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:46551 (local address 127.2.238.131:45175)
0811 04:36:27.166670 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.166675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.167000 (+ 325us) server_negotiation.cc:407] Connection header received
0811 04:36:27.167215 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.167220 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.167279 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.167355 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:27.167608 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.168469 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.170311 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.170993 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.171133 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.171194 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.171581 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.171644 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:36:27.171816 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.166173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45175 (local address 127.2.238.130:46551)
0811 04:36:27.166909 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:27.166925 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:27.167060 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:27.167359 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:27.167362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:27.167375 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:27.167459 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.167465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.168616 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.168619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.170129 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:27.170144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.171327 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.171332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:27.171457 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.171487 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 04:36:27.171565 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:36:27.171644 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":164,"threads_started":1}
I20260811 04:36:27.172609 3783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d433b6451894b738b725d3b2c719607" candidate_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1212fb30ab954976b55aeea7cfabf29b" is_pre_election: true
I20260811 04:36:27.172611 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d433b6451894b738b725d3b2c719607" candidate_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2b17d943774fecb830f612fb58025b" is_pre_election: true
I20260811 04:36:27.172931 3783 raft_consensus.cc:2454] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff3ee9fe58e4696adba9d9d95b6f32a in term 0.
I20260811 04:36:27.172936 4078 raft_consensus.cc:2454] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff3ee9fe58e4696adba9d9d95b6f32a in term 0.
I20260811 04:36:27.173448 3856 leader_election.cc:302] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1212fb30ab954976b55aeea7cfabf29b, dff3ee9fe58e4696adba9d9d95b6f32a; no voters:
I20260811 04:36:27.173731 4156 raft_consensus.cc:2781] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:27.173837 4156 raft_consensus.cc:482] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:27.173883 4156 raft_consensus.cc:3037] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:27.175024 4156 raft_consensus.cc:504] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.175374 4156 leader_election.cc:288] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [CANDIDATE]: Term 1 election: Requested vote from peers 1212fb30ab954976b55aeea7cfabf29b (127.2.238.129:41857), ce2b17d943774fecb830f612fb58025b (127.2.238.131:45175)
I20260811 04:36:27.175737 3783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d433b6451894b738b725d3b2c719607" candidate_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1212fb30ab954976b55aeea7cfabf29b"
I20260811 04:36:27.175803 4078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d433b6451894b738b725d3b2c719607" candidate_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce2b17d943774fecb830f612fb58025b"
I20260811 04:36:27.175872 3783 raft_consensus.cc:3037] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:27.175909 4078 raft_consensus.cc:3037] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:27.177223 4078 raft_consensus.cc:2454] T 1d433b6451894b738b725d3b2c719607 P ce2b17d943774fecb830f612fb58025b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff3ee9fe58e4696adba9d9d95b6f32a in term 1.
I20260811 04:36:27.177281 3783 raft_consensus.cc:2454] T 1d433b6451894b738b725d3b2c719607 P 1212fb30ab954976b55aeea7cfabf29b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff3ee9fe58e4696adba9d9d95b6f32a in term 1.
I20260811 04:36:27.177542 3856 leader_election.cc:302] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1212fb30ab954976b55aeea7cfabf29b, dff3ee9fe58e4696adba9d9d95b6f32a; no voters:
I20260811 04:36:27.177837 4156 raft_consensus.cc:2781] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:27.178094 4156 raft_consensus.cc:686] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [term 1 LEADER]: Becoming Leader. State: Replica: dff3ee9fe58e4696adba9d9d95b6f32a, State: Running, Role: LEADER
I20260811 04:36:27.178474 4156 consensus_queue.cc:237] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } }
I20260811 04:36:27.181682 3617 catalog_manager.cc:5495] T 1d433b6451894b738b725d3b2c719607 P dff3ee9fe58e4696adba9d9d95b6f32a reported cstate change: term changed from 0 to 1, leader changed from <none> to dff3ee9fe58e4696adba9d9d95b6f32a (127.2.238.130). New cstate: current_term: 1 leader_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1212fb30ab954976b55aeea7cfabf29b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce2b17d943774fecb830f612fb58025b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dff3ee9fe58e4696adba9d9d95b6f32a" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34527 } health_report { overall_health: HEALTHY } } }
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:27.216910 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.200765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.2.238.129:41857)
0811 04:36:27.200935 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.200940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.201128 (+ 188us) server_negotiation.cc:407] Connection header received
0811 04:36:27.201311 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.201314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.201370 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.201460 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:27.202267 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.202790 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.203641 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.203877 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.203993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.206847 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:27.206863 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:27.206880 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:27.206910 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:27.214822 (+ 7912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.215419 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.215427 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.215428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.215527 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.215857 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.215861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.215865 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.216162 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:27.216408 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.216583 (+ 175us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.216723 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:27.245877 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.236994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44786 (local address 127.2.238.129:41857)
0811 04:36:27.237152 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.237156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.237295 (+ 139us) server_negotiation.cc:407] Connection header received
0811 04:36:27.237411 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.237413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.237459 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.237566 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:27.238355 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.238886 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.239727 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.239982 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.240098 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.242577 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:27.242580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:27.242582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:27.242607 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:27.244036 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.244517 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.244520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.244522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.244635 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.244916 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.244919 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.244921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.245099 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:27.245234 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.245529 (+ 295us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.245677 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[3570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:27.274559 4143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:27.265631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44788 (local address 127.2.238.129:41857)
0811 04:36:27.265831 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:27.265835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:27.266026 (+ 191us) server_negotiation.cc:407] Connection header received
0811 04:36:27.266182 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:27.266184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:27.266230 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:27.266302 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:27.267239 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.267712 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.268786 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:27.269004 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:27.269087 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:27.271541 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:27.271544 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:27.271546 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:27.271569 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:27.272944 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.273319 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.273323 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.273325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.273433 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:27.273687 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:27.273690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:27.273692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:27.273947 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:27.274067 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:27.274288 (+ 221us) server_negotiation.cc:299] Negotiation successful
0811 04:36:27.274408 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260811 04:36:27.276290 3738 tablet_service.cc:2908] Rejecting scan request for tablet 1d433b6451894b738b725d3b2c719607: Uninitialized: clean time has not yet been initialized
I20260811 04:36:27.278852 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3689
I20260811 04:36:27.288440 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3837
I20260811 04:36:27.296007 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3984
I20260811 04:36:27.304351 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 3584
2026-08-11T04:36:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2417 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[4176](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[4176](info): set up 2 sockets
Aug 11 04:36:27 dist-test-slave-0kls krb5kdc[4176](info): commencing operation
krb5kdc: starting...
W20260811 04:36:28.373776 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:28 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:28.504868 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34837
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:28.620826 4191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:28.621122 4191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:28.621166 4191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:28.625552 4191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:28.625662 4191 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 04:36:28.625703 4191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:28.625730 4191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:28.625766 4191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:28.625811 4191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:28.631494 4191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34837
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:28.632648 4191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:28.633728 4191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:28.642390 4199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:28.642393 4196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:28.642441 4197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:28.642560 4191 server_base.cc:1034] running on GCE node
I20260811 04:36:28.643294 4191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:28.644426 4191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:28.645614 4191 hybrid_clock.cc:648] HybridClock initialized: now 1786422988645582 us; error 45 us; skew 500 ppm
Aug 11 04:36:28 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:28.651299 4191 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:28.652204 4191 webserver.cc:460] Webserver started at http://127.2.238.190:39725/ using document root <none> and password file <none>
I20260811 04:36:28.652801 4191 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:28.652891 4191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:28.653120 4191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:28.655331 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "41b65340a2d3427bb2c1b9963c047a28"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "7fa9b260ea6d29cfdac9c433a2959f9b"
server_key_iv: "cc08f9fb6ea04e5b0fffd037735a2a85"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.655922 4191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "41b65340a2d3427bb2c1b9963c047a28"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "7fa9b260ea6d29cfdac9c433a2959f9b"
server_key_iv: "cc08f9fb6ea04e5b0fffd037735a2a85"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.659895 4191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:28.662425 4206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:28.663708 4191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:28.663878 4191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "41b65340a2d3427bb2c1b9963c047a28"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "7fa9b260ea6d29cfdac9c433a2959f9b"
server_key_iv: "cc08f9fb6ea04e5b0fffd037735a2a85"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.664002 4191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:28.675415 4191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:28.678647 4191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:28.678879 4191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:28.687506 4191 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34837
I20260811 04:36:28.687551 4276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34837 every 8 connection(s)
I20260811 04:36:28.688521 4191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:28.691449 4277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:28.692245 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 4191
I20260811 04:36:28.692363 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:28.692651 3002 external_mini_cluster.cc:1428] Setting key 5583984ac04703e5f0e3ee1988bfb5b1
I20260811 04:36:28.697742 4277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Bootstrap starting.
I20260811 04:36:28.700173 4277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:28.700953 4277 log.cc:824] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:36:28 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:28.703392 4277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: No bootstrap required, opened a new log
I20260811 04:36:28.707211 4277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } }
I20260811 04:36:28.707504 4277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:28.707594 4277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41b65340a2d3427bb2c1b9963c047a28, State: Initialized, Role: FOLLOWER
I20260811 04:36:28.707929 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:28.694207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33940 (local address 127.2.238.190:34837)
0811 04:36:28.694669 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:28.694684 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:28.694719 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:36:28.695364 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:28.695381 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:28.695711 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:28.696054 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:28.696833 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:28.697876 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:28.698732 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:28.699242 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:28.699422 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:28.702335 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:28.702341 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:28.702355 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:28.702383 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:28.704726 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:28.705250 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:28.705255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:28.705261 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:28.705385 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:28.705695 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:28.705697 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:28.705698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:28.706174 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:28.706344 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:28.706644 (+ 300us) server_negotiation.cc:299] Negotiation successful
0811 04:36:28.706961 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":135,"threads_started":1}
I20260811 04:36:28.708107 4277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } }
I20260811 04:36:28.708282 4277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:28.708367 4277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:28.708469 4277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:28.709620 4277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } }
I20260811 04:36:28.709996 4277 leader_election.cc:302] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41b65340a2d3427bb2c1b9963c047a28; no voters:
I20260811 04:36:28.710397 4277 leader_election.cc:288] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:28.710507 4283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:28.710717 4283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [term 1 LEADER]: Becoming Leader. State: Replica: 41b65340a2d3427bb2c1b9963c047a28, State: Running, Role: LEADER
I20260811 04:36:28.711027 4283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } }
I20260811 04:36:28.711634 4277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:28.712721 4284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41b65340a2d3427bb2c1b9963c047a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } } }
I20260811 04:36:28.712869 4284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:28.713268 4293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:28.713238 4285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41b65340a2d3427bb2c1b9963c047a28. Latest consensus state: current_term: 1 leader_uuid: "41b65340a2d3427bb2c1b9963c047a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41b65340a2d3427bb2c1b9963c047a28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34837 } } }
I20260811 04:36:28.713810 4285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:28.716612 4293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:28.722046 4293 catalog_manager.cc:1347] Generated new cluster ID: fead13f302e54a9cb0258b15cec5513c
I20260811 04:36:28.722113 4293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:28.738143 4293 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:28.739123 4293 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:28.742952 4293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Generated new TSK 0
I20260811 04:36:28.743638 4293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:28.781301 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34837
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:28.917450 4309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:28.917819 4309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:28.917909 4309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:28.922142 4309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:28.922219 4309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:28.922260 4309 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:28.922335 4309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:28.928166 4309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:34837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:28.929437 4309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:28.930717 4309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:28.941215 4314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:28.941262 4315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:28.941401 4309 server_base.cc:1034] running on GCE node
W20260811 04:36:28.941215 4318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:28.942018 4309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:28.942713 4309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:28.943914 4309 hybrid_clock.cc:648] HybridClock initialized: now 1786422988943871 us; error 41 us; skew 500 ppm
Aug 11 04:36:28 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:28.947052 4309 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:28.948088 4309 webserver.cc:460] Webserver started at http://127.2.238.129:41623/ using document root <none> and password file <none>
I20260811 04:36:28.948654 4309 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:28.948729 4309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:28.948917 4309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:28.951155 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "907b07892c1b4c51a96a67b52c38fba7"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "f00dadb17e7916bf3ef65125d4daedf2"
server_key_iv: "8e50553e69920b4669a8cb6730844748"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.951759 4309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "907b07892c1b4c51a96a67b52c38fba7"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "f00dadb17e7916bf3ef65125d4daedf2"
server_key_iv: "8e50553e69920b4669a8cb6730844748"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.956053 4309 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:28.958911 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:28.960316 4309 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:28.960498 4309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "907b07892c1b4c51a96a67b52c38fba7"
format_stamp: "Formatted at 2026-08-11 04:36:28 on dist-test-slave-0kls"
server_key: "f00dadb17e7916bf3ef65125d4daedf2"
server_key_iv: "8e50553e69920b4669a8cb6730844748"
server_key_version: "encryptionkey@0"
I20260811 04:36:28.960640 4309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:28.988456 4309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:28.992403 4309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:28.992640 4309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:28.994294 4309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:28.994370 4309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:28.994441 4309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:28.994484 4309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:28 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:29.006558 4309 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:37899
I20260811 04:36:29.006668 4456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:37899 every 8 connection(s)
I20260811 04:36:29.007565 4309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:29.009388 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:28.995334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:37381 (local address 127.2.238.190:34837)
0811 04:36:28.995466 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:28.995470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:28.996676 (+ 1206us) server_negotiation.cc:407] Connection header received
0811 04:36:28.997958 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:28.997961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:28.998059 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:28.998179 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:28.999621 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.000278 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.001491 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.001888 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.002013 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.005199 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.005202 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.005205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.005238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.007069 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.007657 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.007663 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.007666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.007754 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.008186 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.008189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.008190 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.008430 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.008530 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.009039 (+ 509us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.009198 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:36:29.010250 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 4309
I20260811 04:36:29.010377 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:29.010365 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:28.995965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.129:37381)
0811 04:36:28.996499 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:28.996545 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:28.997683 (+ 1138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:28.998377 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:28.998385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:28.998740 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:28.999442 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:28.999463 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.000419 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.000423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.001345 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.001356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.002075 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.002079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.002275 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.002934 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.002953 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.004993 (+ 2040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.007226 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.007233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.007249 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.007526 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.007852 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.007855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.007857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.008010 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.008536 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.008545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.008798 (+ 253us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.008997 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.009354 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":149,"threads_started":1}
I20260811 04:36:29.010876 3002 external_mini_cluster.cc:1428] Setting key da27879b54533c9514dc7b0ffef0c7d8
I20260811 04:36:29.012380 4457 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34837
I20260811 04:36:29.012571 4457 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:29.013170 4457 heartbeater.cc:511] Master 127.2.238.190:34837 requested a full tablet report, sending...
I20260811 04:36:29.015012 4226 ts_manager.cc:194] Registered new tserver with Master: 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899)
I20260811 04:36:29.016220 4226 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:37381
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:29.025005 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.016939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:38333 (local address 127.2.238.190:34837)
0811 04:36:29.017075 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.017082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.017098 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:36:29.017141 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.017146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.017197 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.017296 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.017878 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.018781 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.019622 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.020062 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.020195 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.021365 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.021370 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.021375 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.021409 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.023279 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.023761 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.023765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.023767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.023868 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.024149 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.024151 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.024153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.024355 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.024438 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.024767 (+ 329us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.024880 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:36:29.025107 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.014870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.129:38333)
0811 04:36:29.015165 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.015185 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.015291 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.017333 (+ 2042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.017337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.017586 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.017708 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.017715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.018952 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.018954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.019495 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.019503 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.020202 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.020204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.020277 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.020848 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.020860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.021197 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.023405 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.023408 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.023410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.023621 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.023973 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.023975 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.023977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.024061 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.024448 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.024450 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.024543 (+ 93us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.024720 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.024920 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:29.045022 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34837
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:29.161854 4468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:29.162201 4468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:29.162250 4468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:29.166548 4468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:29.166626 4468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:29.166667 4468 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:29.166700 4468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:29.171993 4468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:34837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:29.173091 4468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:29.174152 4468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:29.184269 4474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:29.184249 4473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:29.184429 4468 server_base.cc:1034] running on GCE node
W20260811 04:36:29.184249 4476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:29.185099 4468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:29.185854 4468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:29.187032 4468 hybrid_clock.cc:648] HybridClock initialized: now 1786422989187017 us; error 27 us; skew 500 ppm
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:29.189983 4468 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:29.190960 4468 webserver.cc:460] Webserver started at http://127.2.238.130:38891/ using document root <none> and password file <none>
I20260811 04:36:29.191555 4468 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:29.191648 4468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:29.191897 4468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:29.194075 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "e745992bb906476b9f178e12c1c6880c"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "7f90d067762a6f4e8933385e0feb165f"
server_key_iv: "8bf0e61680ec77ee13f8bbb72d944147"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.194720 4468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "e745992bb906476b9f178e12c1c6880c"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "7f90d067762a6f4e8933385e0feb165f"
server_key_iv: "8bf0e61680ec77ee13f8bbb72d944147"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.198349 4468 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:36:29.200850 4484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:29.202224 4468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:29.202373 4468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "e745992bb906476b9f178e12c1c6880c"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "7f90d067762a6f4e8933385e0feb165f"
server_key_iv: "8bf0e61680ec77ee13f8bbb72d944147"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.202486 4468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:29.230922 4468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:29.234455 4468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:29.234684 4468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:29.236198 4468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:29.236272 4468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:29.236347 4468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:29.236388 4468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:29.247758 4468 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:40151
I20260811 04:36:29.247882 4621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:40151 every 8 connection(s)
I20260811 04:36:29.248896 4468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:29.251312 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.237114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45023 (local address 127.2.238.190:34837)
0811 04:36:29.237247 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.237251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.238362 (+ 1111us) server_negotiation.cc:407] Connection header received
0811 04:36:29.239457 (+ 1095us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.239461 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.239518 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.239575 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.241498 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.242210 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.244190 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.244567 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.244663 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.247746 (+ 3083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.247752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.247758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.247794 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.249287 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.249891 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.249894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.249895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.249982 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.250301 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.250303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.250304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.250477 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.250546 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.251084 (+ 538us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.251186 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:29.252243 4497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.237671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.130:45023)
0811 04:36:29.238206 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.238251 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.239197 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.240070 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.240103 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.240574 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.241350 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.241373 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.242358 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.242363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.244030 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.244045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.244726 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.244733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.244964 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.245666 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.245694 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.247566 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.249405 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.249410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.249424 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.249774 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.250074 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.250076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.250078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.250199 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.250559 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.250564 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.250816 (+ 252us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.251009 (+ 193us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.251269 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":127,"threads_started":1}
I20260811 04:36:29.253150 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 4468
I20260811 04:36:29.253270 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:29.253549 3002 external_mini_cluster.cc:1428] Setting key 55bafa4d5c004564a319127425c13c75
I20260811 04:36:29.253711 4622 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34837
I20260811 04:36:29.253891 4622 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:29.254477 4622 heartbeater.cc:511] Master 127.2.238.190:34837 requested a full tablet report, sending...
I20260811 04:36:29.255558 4225 ts_manager.cc:194] Registered new tserver with Master: e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151)
I20260811 04:36:29.256390 4225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:45023
I20260811 04:36:29.265143 4630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.257181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.130:45367)
0811 04:36:29.257536 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.257552 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.257690 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.257955 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.257958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.258169 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.258263 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.258269 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.259162 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.259165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.259702 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.259710 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.260389 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.260392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.260462 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.261028 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.261041 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.261380 (+ 339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.263361 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.263364 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.263365 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.263576 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.263926 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.263928 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.263930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.264028 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.264621 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.264624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.264763 (+ 139us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.264857 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.264975 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":97,"threads_started":1}
I20260811 04:36:29.265273 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.256934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45367 (local address 127.2.238.190:34837)
0811 04:36:29.257062 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.257069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.257646 (+ 577us) server_negotiation.cc:407] Connection header received
0811 04:36:29.257770 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.257773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.257847 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.257936 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.258381 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.259019 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.259838 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.260248 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.260386 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.261535 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.261540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.261546 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.261580 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.263242 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.263695 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.263700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.263702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.263800 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.264142 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.264146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.264148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.264485 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.264597 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.264964 (+ 367us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.265095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:29.286216 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34837
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:29.404343 4632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:29.404656 4632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:29.404747 4632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:29.408917 4632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:29.408982 4632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:29.409024 4632 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:29.409055 4632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:29.415081 4632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:34837
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:29.416112 4632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:29.417121 4632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:29.427042 4639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:29.427019 4637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:29.426985 4641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:29.427268 4632 server_base.cc:1034] running on GCE node
I20260811 04:36:29.427772 4632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:29.428443 4632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:29.429632 4632 hybrid_clock.cc:648] HybridClock initialized: now 1786422989429591 us; error 58 us; skew 500 ppm
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:29.432395 4632 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:29.433362 4632 webserver.cc:460] Webserver started at http://127.2.238.131:45083/ using document root <none> and password file <none>
I20260811 04:36:29.433954 4632 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:29.434021 4632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:29.434193 4632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:29.436151 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "66d0254e80548f6e7021e9c5f0c4f4e8"
server_key_iv: "22155cceed96f93ae82daed46945fffe"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.436684 4632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "66d0254e80548f6e7021e9c5f0c4f4e8"
server_key_iv: "22155cceed96f93ae82daed46945fffe"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.440475 4632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:29.442915 4650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:29.444115 4632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:36:29.444248 4632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1"
format_stamp: "Formatted at 2026-08-11 04:36:29 on dist-test-slave-0kls"
server_key: "66d0254e80548f6e7021e9c5f0c4f4e8"
server_key_iv: "22155cceed96f93ae82daed46945fffe"
server_key_version: "encryptionkey@0"
I20260811 04:36:29.444352 4632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:29.467566 4632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:29.470912 4632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:29.471144 4632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:29.472744 4632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:29.472796 4632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:29.472838 4632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:29.472895 4632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:29.484396 4632 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:40025
I20260811 04:36:29.484831 4778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:40025 every 8 connection(s)
I20260811 04:36:29.485422 4632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:29.488277 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.473874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:43867 (local address 127.2.238.190:34837)
0811 04:36:29.474008 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.474012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.475170 (+ 1158us) server_negotiation.cc:407] Connection header received
0811 04:36:29.476484 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.476488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.476559 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.476670 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.478424 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.479374 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.480601 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.480989 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.481097 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.484069 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.484074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.484077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.484122 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.485913 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.486567 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.486571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.486573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.486674 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.487043 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.487046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.487047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.487258 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.487337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.488003 (+ 666us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.488128 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 04:36:29.489151 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.474470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.131:43867)
0811 04:36:29.475007 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.475054 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.476208 (+ 1154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.476902 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.476911 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.477420 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.478235 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.478258 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.479525 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.479529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.480454 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.480463 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.481161 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.481166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.481350 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.482028 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.482054 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.483858 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.486046 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.486052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.486070 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.486436 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.486784 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.486787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.486788 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.486931 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.487351 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.487356 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.487595 (+ 239us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.487898 (+ 303us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.488195 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":160,"threads_started":1}
I20260811 04:36:29.490505 4779 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34837
I20260811 04:36:29.490662 4779 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:29.491315 4779 heartbeater.cc:511] Master 127.2.238.190:34837 requested a full tablet report, sending...
I20260811 04:36:29.492491 4225 ts_manager.cc:194] Registered new tserver with Master: 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025)
I20260811 04:36:29.493240 4225 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:43867
I20260811 04:36:29.494001 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 4632
I20260811 04:36:29.494144 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:29.494426 3002 external_mini_cluster.cc:1428] Setting key 4cfa0f64aa7ea5445a0bc3efdaeedec2
I20260811 04:36:29.496516 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:36:29.501473 4785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.492966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.2.238.131:58683)
0811 04:36:29.493297 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.493316 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.493459 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.493752 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.493755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.494041 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.494267 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.494275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.495340 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.495343 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.496037 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.496046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.496528 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.496531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.496624 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.497182 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.497201 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.497530 (+ 329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.499656 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.499659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.499661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.499901 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.500267 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.500269 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.500271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.500357 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.500968 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.500970 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.501066 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.501172 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.501310 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":26,"thread_start_us":107,"threads_started":1}
I20260811 04:36:29.501664 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.493101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:58683 (local address 127.2.238.190:34837)
0811 04:36:29.493234 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.493239 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.493388 (+ 149us) server_negotiation.cc:407] Connection header received
0811 04:36:29.493574 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.493578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.493628 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.493732 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.494412 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.495155 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.496165 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.496404 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.496523 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.497684 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.497696 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.497698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.497731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.499525 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.500011 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.500017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.500022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.500130 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.500454 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.500460 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.500465 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.500740 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.500862 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.501273 (+ 411us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.501426 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:29.527714 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.519109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.2.238.190:34837)
0811 04:36:29.519251 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.519254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.519266 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:36:29.519308 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.519310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.519350 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.519445 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.520208 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.520738 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.521627 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.521908 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.522023 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.524453 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.524456 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.524459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.524495 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.526025 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.526448 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.526452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.526454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.526540 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.526821 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.526825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.526827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.527016 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.527169 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.527416 (+ 247us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.527529 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 04:36:29.531031 4225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:29.534096 4225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:29.548048 4799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.542551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40151 (local address 127.0.0.1:52660)
0811 04:36:29.542924 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.542971 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.543097 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.543691 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.543697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.543724 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.543869 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.543874 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.545324 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.545330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.546280 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.546288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.547385 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.547388 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.547550 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.547603 (+ 53us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.547738 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.547881 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":5,"thread_start_us":120,"threads_started":1}
I20260811 04:36:29.548112 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.542721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.2.238.130:40151)
0811 04:36:29.543045 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.543058 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.543085 (+ 27us) server_negotiation.cc:407] Connection header received
0811 04:36:29.543259 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.543268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.543495 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.544031 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.544175 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.545201 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.546406 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.547258 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.547374 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.547548 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.547852 (+ 304us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.547954 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":127,"threads_started":1}
I20260811 04:36:29.549120 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.543325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37899 (local address 127.0.0.1:47726)
0811 04:36:29.543555 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.543571 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.543652 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.544765 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.544769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.544788 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.544881 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.544886 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.546266 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.546270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.547517 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.547528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.548646 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.548651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.548783 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.548813 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.548896 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.548945 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":77,"thread_start_us":100,"threads_started":1}
I20260811 04:36:29.549120 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.543651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40025 (local address 127.0.0.1:35988)
0811 04:36:29.543906 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.543919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.544000 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.545069 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.545072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.545085 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.545163 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.545167 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.546586 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.546589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.547682 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.547708 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.548691 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.548694 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.548798 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.548816 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.548879 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.548945 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":40,"thread_start_us":99,"threads_started":1}
I20260811 04:36:29.549194 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.543495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47726 (local address 127.2.238.129:37899)
0811 04:36:29.544305 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.544314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.544340 (+ 26us) server_negotiation.cc:407] Connection header received
0811 04:36:29.544427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.544433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.544612 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.544766 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.545005 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.546147 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.547656 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.548511 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.548625 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.548869 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.548928 (+ 59us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.549071 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":85,"threads_started":1}
I20260811 04:36:29.549749 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.544212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35988 (local address 127.2.238.131:40025)
0811 04:36:29.544603 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.544613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.544633 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:36:29.544706 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.544714 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.544881 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.545086 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.545246 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.546309 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.547854 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.548543 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.549257 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.549467 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.549521 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.549607 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":92,"threads_started":1}
I20260811 04:36:29.551884 4381 tablet_service.cc:1467] Processing CreateTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f (DEFAULT_TABLE table=test-table [id=6b76d2f53f884212a61a8784a38d4fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.551841 4545 tablet_service.cc:1467] Processing CreateTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f (DEFAULT_TABLE table=test-table [id=6b76d2f53f884212a61a8784a38d4fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.552148 4703 tablet_service.cc:1467] Processing CreateTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f (DEFAULT_TABLE table=test-table [id=6b76d2f53f884212a61a8784a38d4fd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.552866 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d016a259944ebbb7b5b3f2ca9cac4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.552903 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d016a259944ebbb7b5b3f2ca9cac4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.552970 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14d016a259944ebbb7b5b3f2ca9cac4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.560266 4805 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Bootstrap starting.
I20260811 04:36:29.560771 4806 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Bootstrap starting.
I20260811 04:36:29.561455 4807 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Bootstrap starting.
I20260811 04:36:29.562780 4805 tablet_bootstrap.cc:654] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.563560 4807 tablet_bootstrap.cc:654] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.563684 4806 tablet_bootstrap.cc:654] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.563686 4805 log.cc:824] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:29.564396 4807 log.cc:824] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:29.564532 4806 log.cc:824] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:29.566362 4805 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: No bootstrap required, opened a new log
I20260811 04:36:29.566402 4807 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: No bootstrap required, opened a new log
I20260811 04:36:29.566601 4807 ts_tablet_manager.cc:1397] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260811 04:36:29.566589 4805 ts_tablet_manager.cc:1397] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:36:29.566993 4806 tablet_bootstrap.cc:492] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: No bootstrap required, opened a new log
I20260811 04:36:29.567198 4806 ts_tablet_manager.cc:1397] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 04:36:29.569890 4807 raft_consensus.cc:348] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.570137 4807 raft_consensus.cc:374] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.570199 4807 raft_consensus.cc:729] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495e1e0e98c46d3a68fafabe1f6b3b1, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.570659 4807 consensus_queue.cc:260] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.571113 4805 raft_consensus.cc:348] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.571101 4806 raft_consensus.cc:348] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.571350 4806 raft_consensus.cc:374] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.571350 4805 raft_consensus.cc:374] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.571401 4806 raft_consensus.cc:729] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e745992bb906476b9f178e12c1c6880c, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.571412 4805 raft_consensus.cc:729] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907b07892c1b4c51a96a67b52c38fba7, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.571681 4807 ts_tablet_manager.cc:1428] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 04:36:29.571859 4805 consensus_queue.cc:260] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.571859 4806 consensus_queue.cc:260] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.572034 4779 heartbeater.cc:503] Master 127.2.238.190:34837 was elected leader, sending a full tablet report...
I20260811 04:36:29.572788 4806 ts_tablet_manager.cc:1428] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:36:29.572788 4805 ts_tablet_manager.cc:1428] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:36:29.573134 4457 heartbeater.cc:503] Master 127.2.238.190:34837 was elected leader, sending a full tablet report...
I20260811 04:36:29.573154 4622 heartbeater.cc:503] Master 127.2.238.190:34837 was elected leader, sending a full tablet report...
I20260811 04:36:29.616747 4813 raft_consensus.cc:482] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:29.617018 4813 raft_consensus.cc:504] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.618432 4813 leader_election.cc:288] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151), 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899)
I20260811 04:36:29.622620 4659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.618726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40151 (local address 127.2.238.131:53289)
0811 04:36:29.618875 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.618893 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.618996 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.619379 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.619382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.619402 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.619514 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.619523 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.620564 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.620570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.621405 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.621414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.622160 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.622164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.622299 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.622318 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.622410 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.622483 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":24}
I20260811 04:36:29.622620 4800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.618852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53289 (local address 127.2.238.130:40151)
0811 04:36:29.618998 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.619004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.619026 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:36:29.619089 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.619093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.619149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.619244 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.619656 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.620456 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.621541 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.622038 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.622178 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.622241 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.622427 (+ 186us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.622496 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:29.623179 4816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.619014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37899 (local address 127.2.238.131:53581)
0811 04:36:29.619502 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.619515 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.619619 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.619899 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.619902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.619921 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.620035 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.620041 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.620809 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.620814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.621991 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.622001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.622694 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.622698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.622797 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.622825 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.622939 (+ 114us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.623006 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":3,"thread_start_us":233,"threads_started":1}
I20260811 04:36:29.623208 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.619174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53581 (local address 127.2.238.129:37899)
0811 04:36:29.619305 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.619310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.619614 (+ 304us) server_negotiation.cc:407] Connection header received
0811 04:36:29.619711 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.619714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.619769 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.619863 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.620147 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.620684 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.622130 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.622556 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.622689 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.622748 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.622971 (+ 223us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.623058 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:36:29.623364 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d016a259944ebbb7b5b3f2ca9cac4f" candidate_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e745992bb906476b9f178e12c1c6880c" is_pre_election: true
I20260811 04:36:29.623697 4567 raft_consensus.cc:2454] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2495e1e0e98c46d3a68fafabe1f6b3b1 in term 0.
I20260811 04:36:29.624162 4406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d016a259944ebbb7b5b3f2ca9cac4f" candidate_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907b07892c1b4c51a96a67b52c38fba7" is_pre_election: true
I20260811 04:36:29.624228 4654 leader_election.cc:302] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495e1e0e98c46d3a68fafabe1f6b3b1, e745992bb906476b9f178e12c1c6880c; no voters:
I20260811 04:36:29.624434 4406 raft_consensus.cc:2454] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2495e1e0e98c46d3a68fafabe1f6b3b1 in term 0.
I20260811 04:36:29.624521 4813 raft_consensus.cc:2781] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:29.624600 4813 raft_consensus.cc:482] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:29.624655 4813 raft_consensus.cc:3037] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.625765 4813 raft_consensus.cc:504] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.626134 4813 leader_election.cc:288] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [CANDIDATE]: Term 1 election: Requested vote from peers e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151), 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899)
I20260811 04:36:29.626531 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d016a259944ebbb7b5b3f2ca9cac4f" candidate_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e745992bb906476b9f178e12c1c6880c"
I20260811 04:36:29.626654 4406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14d016a259944ebbb7b5b3f2ca9cac4f" candidate_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907b07892c1b4c51a96a67b52c38fba7"
I20260811 04:36:29.626648 4567 raft_consensus.cc:3037] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.626804 4406 raft_consensus.cc:3037] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.627882 4406 raft_consensus.cc:2454] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2495e1e0e98c46d3a68fafabe1f6b3b1 in term 1.
I20260811 04:36:29.627954 4567 raft_consensus.cc:2454] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2495e1e0e98c46d3a68fafabe1f6b3b1 in term 1.
I20260811 04:36:29.628242 4654 leader_election.cc:302] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495e1e0e98c46d3a68fafabe1f6b3b1, 907b07892c1b4c51a96a67b52c38fba7; no voters:
I20260811 04:36:29.628410 4813 raft_consensus.cc:2781] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:29.628639 4813 raft_consensus.cc:686] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 LEADER]: Becoming Leader. State: Replica: 2495e1e0e98c46d3a68fafabe1f6b3b1, State: Running, Role: LEADER
I20260811 04:36:29.628978 4813 consensus_queue.cc:237] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.632350 4226 catalog_manager.cc:5495] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } health_report { overall_health: HEALTHY } } }
I20260811 04:36:29.652554 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.649035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35990 (local address 127.2.238.131:40025)
0811 04:36:29.649182 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.649185 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.649198 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:29.649279 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.649281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.649332 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.649423 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:29.649675 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.650254 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.651163 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.651412 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.651504 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.651804 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:29.652212 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:29.652301 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.652388 (+ 87us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.652439 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 04:36:29.658638 4567 raft_consensus.cc:1261] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Refusing update from remote peer 2495e1e0e98c46d3a68fafabe1f6b3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:29.658697 4406 raft_consensus.cc:1261] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Refusing update from remote peer 2495e1e0e98c46d3a68fafabe1f6b3b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:29.659371 4813 consensus_queue.cc:1035] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:29.659595 4817 consensus_queue.cc:1035] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:29.682200 4225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:29.682534 4225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:29.685375 4225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Sending DeleteTablet for 3 replicas of tablet 14d016a259944ebbb7b5b3f2ca9cac4f
I20260811 04:36:29.686358 4545 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:29 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52660
I20260811 04:36:29.686411 4381 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:29 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:47726
I20260811 04:36:29.686575 4703 tablet_service.cc:1514] Processing DeleteTablet for tablet 14d016a259944ebbb7b5b3f2ca9cac4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:29 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:35988
I20260811 04:36:29.687021 4833 tablet_replica.cc:330] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: stopping tablet replica
I20260811 04:36:29.687434 4833 raft_consensus.cc:2229] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:29.687819 4833 raft_consensus.cc:2258] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:29.688197 4832 tablet_replica.cc:330] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: stopping tablet replica
I20260811 04:36:29.688522 4832 raft_consensus.cc:2229] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:29.688856 4834 tablet_replica.cc:330] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: stopping tablet replica
I20260811 04:36:29.689189 4834 raft_consensus.cc:2229] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:29.689678 4834 raft_consensus.cc:2258] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:29.690021 4833 ts_tablet_manager.cc:1905] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:29.690356 4832 raft_consensus.cc:2258] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:29.691062 4225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:29.691540 4225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:29.693234 4833 ts_tablet_manager.cc:1918] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:29.693359 4833 log.cc:1196] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/14d016a259944ebbb7b5b3f2ca9cac4f
I20260811 04:36:29.693750 4833 ts_tablet_manager.cc:1939] T 14d016a259944ebbb7b5b3f2ca9cac4f P 907b07892c1b4c51a96a67b52c38fba7: Deleting consensus metadata
I20260811 04:36:29.694410 4834 ts_tablet_manager.cc:1905] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:29.696172 4212 catalog_manager.cc:4841] TS 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899): tablet 14d016a259944ebbb7b5b3f2ca9cac4f (table test-table [id=6b76d2f53f884212a61a8784a38d4fd7]) successfully deleted
I20260811 04:36:29.696566 4832 ts_tablet_manager.cc:1905] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:29.698086 4834 ts_tablet_manager.cc:1918] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:29.698258 4834 log.cc:1196] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/14d016a259944ebbb7b5b3f2ca9cac4f
I20260811 04:36:29.698632 4834 ts_tablet_manager.cc:1939] T 14d016a259944ebbb7b5b3f2ca9cac4f P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting consensus metadata
I20260811 04:36:29.699380 4832 ts_tablet_manager.cc:1918] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:29.699487 4545 tablet_service.cc:1467] Processing CreateTablet for tablet 9adce377d4e0462d880b88c399749262 (DEFAULT_TABLE table=test-table [id=2691b3d203f3463abe670cb9b28d0d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.699822 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adce377d4e0462d880b88c399749262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.699752 4703 tablet_service.cc:1467] Processing CreateTablet for tablet 9adce377d4e0462d880b88c399749262 (DEFAULT_TABLE table=test-table [id=2691b3d203f3463abe670cb9b28d0d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.700016 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adce377d4e0462d880b88c399749262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.700580 4381 tablet_service.cc:1467] Processing CreateTablet for tablet 9adce377d4e0462d880b88c399749262 (DEFAULT_TABLE table=test-table [id=2691b3d203f3463abe670cb9b28d0d56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:29.700841 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9adce377d4e0462d880b88c399749262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.702000 4212 catalog_manager.cc:4841] TS 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025): tablet 14d016a259944ebbb7b5b3f2ca9cac4f (table test-table [id=6b76d2f53f884212a61a8784a38d4fd7]) successfully deleted
I20260811 04:36:29.702396 4807 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Bootstrap starting.
I20260811 04:36:29.702446 4806 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Bootstrap starting.
I20260811 04:36:29.699487 4832 log.cc:1196] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/14d016a259944ebbb7b5b3f2ca9cac4f
I20260811 04:36:29.703188 4832 ts_tablet_manager.cc:1939] T 14d016a259944ebbb7b5b3f2ca9cac4f P e745992bb906476b9f178e12c1c6880c: Deleting consensus metadata
I20260811 04:36:29.703269 4805 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Bootstrap starting.
I20260811 04:36:29.703482 4807 tablet_bootstrap.cc:654] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.703517 4806 tablet_bootstrap.cc:654] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.704360 4805 tablet_bootstrap.cc:654] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.704537 4212 catalog_manager.cc:4841] TS e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151): tablet 14d016a259944ebbb7b5b3f2ca9cac4f (table test-table [id=6b76d2f53f884212a61a8784a38d4fd7]) successfully deleted
I20260811 04:36:29.704856 4807 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: No bootstrap required, opened a new log
I20260811 04:36:29.704941 4807 ts_tablet_manager.cc:1397] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:29.705453 4806 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: No bootstrap required, opened a new log
I20260811 04:36:29.705567 4806 ts_tablet_manager.cc:1397] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:29.705600 4807 raft_consensus.cc:348] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.705730 4807 raft_consensus.cc:374] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.705789 4807 raft_consensus.cc:729] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495e1e0e98c46d3a68fafabe1f6b3b1, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.705945 4807 consensus_queue.cc:260] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.706130 4806 raft_consensus.cc:348] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.706245 4807 ts_tablet_manager.cc:1428] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:29.706261 4806 raft_consensus.cc:374] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.706296 4806 raft_consensus.cc:729] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e745992bb906476b9f178e12c1c6880c, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.706423 4806 consensus_queue.cc:260] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.706686 4806 ts_tablet_manager.cc:1428] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:29.707906 4805 tablet_bootstrap.cc:492] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: No bootstrap required, opened a new log
I20260811 04:36:29.708004 4805 ts_tablet_manager.cc:1397] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260811 04:36:29.708915 4805 raft_consensus.cc:348] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.709074 4805 raft_consensus.cc:374] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.709131 4805 raft_consensus.cc:729] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907b07892c1b4c51a96a67b52c38fba7, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.709299 4805 consensus_queue.cc:260] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.709578 4805 ts_tablet_manager.cc:1428] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260811 04:36:29.737151 4780 tablet.cc:2369] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:36:29.750481 4623 tablet.cc:2369] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:36:29.759191 4458 tablet.cc:2369] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:36:29.811234 4815 raft_consensus.cc:482] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:29.811409 4815 raft_consensus.cc:504] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.812743 4815 leader_election.cc:288] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899), 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025)
I20260811 04:36:29.816998 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.813768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:56333 (local address 127.2.238.131:40025)
0811 04:36:29.813914 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.813918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.813931 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:29.814033 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.814036 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.814088 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.814171 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.814423 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.814891 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.815895 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.816420 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.816549 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.816612 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.816807 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.816870 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 04:36:29.817009 4839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.813384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40025 (local address 127.2.238.130:56333)
0811 04:36:29.813828 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.813843 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.813928 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.814182 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.814185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.814200 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.814312 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.814317 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.815002 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.815007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.815773 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.815781 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.816549 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.816553 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.816683 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.816703 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.816790 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.816862 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":70,"threads_started":1}
I20260811 04:36:29.817970 4838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.812942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37899 (local address 127.2.238.130:48835)
0811 04:36:29.813341 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.813355 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.813470 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.813969 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.813972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.813995 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:29.814114 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.814122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.815188 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.815191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.816411 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.816423 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.817500 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.817503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.817636 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.817661 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.817735 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.817828 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":137,"threads_started":1}
I20260811 04:36:29.817994 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.813128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:48835 (local address 127.2.238.129:37899)
0811 04:36:29.813317 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.813323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.813410 (+ 87us) server_negotiation.cc:407] Connection header received
0811 04:36:29.813646 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.813649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.813708 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.813827 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:29.814258 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.814991 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.816599 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.817239 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.817351 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.817400 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.817761 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.817871 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 04:36:29.817924 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adce377d4e0462d880b88c399749262" candidate_uuid: "e745992bb906476b9f178e12c1c6880c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" is_pre_election: true
I20260811 04:36:29.818220 4725 raft_consensus.cc:2454] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e745992bb906476b9f178e12c1c6880c in term 0.
I20260811 04:36:29.818548 4406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adce377d4e0462d880b88c399749262" candidate_uuid: "e745992bb906476b9f178e12c1c6880c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907b07892c1b4c51a96a67b52c38fba7" is_pre_election: true
I20260811 04:36:29.818678 4406 raft_consensus.cc:2454] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e745992bb906476b9f178e12c1c6880c in term 0.
I20260811 04:36:29.818750 4492 leader_election.cc:302] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495e1e0e98c46d3a68fafabe1f6b3b1, e745992bb906476b9f178e12c1c6880c; no voters:
I20260811 04:36:29.819061 4815 raft_consensus.cc:2781] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:29.819140 4815 raft_consensus.cc:482] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:29.819188 4815 raft_consensus.cc:3037] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.820283 4815 raft_consensus.cc:504] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.820645 4815 leader_election.cc:288] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [CANDIDATE]: Term 1 election: Requested vote from peers 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899), 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025)
I20260811 04:36:29.821084 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adce377d4e0462d880b88c399749262" candidate_uuid: "e745992bb906476b9f178e12c1c6880c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1"
I20260811 04:36:29.821101 4406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9adce377d4e0462d880b88c399749262" candidate_uuid: "e745992bb906476b9f178e12c1c6880c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "907b07892c1b4c51a96a67b52c38fba7"
I20260811 04:36:29.821199 4406 raft_consensus.cc:3037] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.821199 4725 raft_consensus.cc:3037] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:29.822350 4725 raft_consensus.cc:2454] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e745992bb906476b9f178e12c1c6880c in term 1.
I20260811 04:36:29.822350 4406 raft_consensus.cc:2454] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e745992bb906476b9f178e12c1c6880c in term 1.
I20260811 04:36:29.822654 4492 leader_election.cc:302] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 907b07892c1b4c51a96a67b52c38fba7, e745992bb906476b9f178e12c1c6880c; no voters:
I20260811 04:36:29.822948 4815 raft_consensus.cc:2781] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:29.823143 4815 raft_consensus.cc:686] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 1 LEADER]: Becoming Leader. State: Replica: e745992bb906476b9f178e12c1c6880c, State: Running, Role: LEADER
I20260811 04:36:29.823414 4815 consensus_queue.cc:237] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.826072 4224 catalog_manager.cc:5495] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c reported cstate change: term changed from 0 to 1, leader changed from <none> to e745992bb906476b9f178e12c1c6880c (127.2.238.130). New cstate: current_term: 1 leader_uuid: "e745992bb906476b9f178e12c1c6880c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } health_report { overall_health: UNKNOWN } } }
Aug 11 04:36:29 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422988, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:29.888463 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.879947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.2.238.190:34837)
0811 04:36:29.880152 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:29.880157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:29.880335 (+ 178us) server_negotiation.cc:407] Connection header received
0811 04:36:29.880524 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:29.880526 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:29.880565 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:29.880656 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:29.881359 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.881972 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.882758 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.882974 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.883082 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.885457 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:29.885460 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:29.885463 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:29.885487 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:29.886757 (+ 1270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.887211 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.887214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.887215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.887287 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:29.887595 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:29.887597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:29.887599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:29.887795 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:29.887873 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:29.888193 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 04:36:29.888324 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":67}
I20260811 04:36:29.888466 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:29.879844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34837 (local address 127.0.0.1:33964)
0811 04:36:29.880239 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:29.880253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:29.880406 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:29.880674 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:29.880677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:29.881137 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:29.881238 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.881244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.882075 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.882088 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.882638 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:29.882646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:29.883098 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:29.883101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:29.883184 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:29.883791 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:29.883804 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:29.885308 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:29.886868 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.886877 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.886883 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.887124 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.887395 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:29.887398 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:29.887399 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:29.887504 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:29.887892 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:29.887897 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:29.888021 (+ 124us) client_negotiation.cc:769] Sending connection context
0811 04:36:29.888133 (+ 112us) client_negotiation.cc:240] Negotiation successful
0811 04:36:29.888268 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260811 04:36:29.892992 4224 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:33964:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 04:36:29.893901 4224 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:29.899150 4545 tablet_service.cc:1467] Processing CreateTablet for tablet 467d39db9dc34b60b7ef6aa4b09e9d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db1516b4f3e94b64a505c3bb7d50e689]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:29.899150 4381 tablet_service.cc:1467] Processing CreateTablet for tablet 467d39db9dc34b60b7ef6aa4b09e9d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db1516b4f3e94b64a505c3bb7d50e689]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:29.899360 4703 tablet_service.cc:1467] Processing CreateTablet for tablet 467d39db9dc34b60b7ef6aa4b09e9d00 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=db1516b4f3e94b64a505c3bb7d50e689]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:29.899580 4381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467d39db9dc34b60b7ef6aa4b09e9d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.899578 4545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467d39db9dc34b60b7ef6aa4b09e9d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.899806 4703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 467d39db9dc34b60b7ef6aa4b09e9d00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:29.902483 4807 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Bootstrap starting.
I20260811 04:36:29.902568 4806 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: Bootstrap starting.
I20260811 04:36:29.902568 4805 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: Bootstrap starting.
I20260811 04:36:29.903280 4807 tablet_bootstrap.cc:654] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.903565 4806 tablet_bootstrap.cc:654] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.903596 4805 tablet_bootstrap.cc:654] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:29.905010 4807 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: No bootstrap required, opened a new log
I20260811 04:36:29.905025 4806 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: No bootstrap required, opened a new log
I20260811 04:36:29.905115 4806 ts_tablet_manager.cc:1397] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 04:36:29.905117 4807 ts_tablet_manager.cc:1397] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260811 04:36:29.905222 4805 tablet_bootstrap.cc:492] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: No bootstrap required, opened a new log
I20260811 04:36:29.905323 4805 ts_tablet_manager.cc:1397] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:29.905591 4807 raft_consensus.cc:348] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.905740 4807 raft_consensus.cc:374] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.905697 4806 raft_consensus.cc:348] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.905802 4807 raft_consensus.cc:729] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2495e1e0e98c46d3a68fafabe1f6b3b1, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.905856 4806 raft_consensus.cc:374] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.905771 4805 raft_consensus.cc:348] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.905894 4806 raft_consensus.cc:729] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e745992bb906476b9f178e12c1c6880c, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.905910 4805 raft_consensus.cc:374] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:29.905946 4805 raft_consensus.cc:729] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907b07892c1b4c51a96a67b52c38fba7, State: Initialized, Role: FOLLOWER
I20260811 04:36:29.905944 4807 consensus_queue.cc:260] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.906054 4806 consensus_queue.cc:260] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.906086 4805 consensus_queue.cc:260] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:29.906263 4807 ts_tablet_manager.cc:1428] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260811 04:36:29.906333 4806 ts_tablet_manager.cc:1428] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:29.906378 4805 ts_tablet_manager.cc:1428] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:29.906663 4817 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:29.906772 4817 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:29.906898 4815 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:29.907035 4815 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:29.907006 4814 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:29.907153 4814 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:30.174129 4814 raft_consensus.cc:482] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:30.174283 4814 raft_consensus.cc:504] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:30.175436 4814 leader_election.cc:288] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151), 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025)
I20260811 04:36:30.179879 4804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.176348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:54479 (local address 127.2.238.131:40025)
0811 04:36:30.176480 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.176484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.176593 (+ 109us) server_negotiation.cc:407] Connection header received
0811 04:36:30.176716 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.176719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.176767 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.176851 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:30.177090 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.177605 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.178898 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.179296 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.179452 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.179500 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.179684 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.179759 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:36:30.179903 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.175780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40151 (local address 127.2.238.129:34439)
0811 04:36:30.176020 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:30.176034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:30.176174 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:30.176543 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:30.176546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:30.176586 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:30.176737 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.176744 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.177594 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.177598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.178699 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.178709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.179401 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.179407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.179492 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.179523 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 04:36:30.179595 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 04:36:30.179715 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":46,"thread_start_us":78,"threads_started":1}
I20260811 04:36:30.179899 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.175866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:34439 (local address 127.2.238.130:40151)
0811 04:36:30.176140 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.176145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.176164 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:36:30.176309 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.176312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.176366 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.176478 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:30.176882 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.177483 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.178822 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.179248 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.179376 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.179457 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.179645 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.179743 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":61,"threads_started":1}
I20260811 04:36:30.179915 4854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.176170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40025 (local address 127.2.238.129:54479)
0811 04:36:30.176506 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:30.176521 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:30.176611 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:30.176875 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:30.176877 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:30.176895 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:30.176985 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.176991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.177748 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.177751 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.178744 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.178774 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.179460 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.179465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.179543 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.179561 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:36:30.179653 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 04:36:30.179767 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":78,"thread_start_us":86,"threads_started":1}
I20260811 04:36:30.180398 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467d39db9dc34b60b7ef6aa4b09e9d00" candidate_uuid: "907b07892c1b4c51a96a67b52c38fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e745992bb906476b9f178e12c1c6880c" is_pre_election: true
I20260811 04:36:30.180423 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467d39db9dc34b60b7ef6aa4b09e9d00" candidate_uuid: "907b07892c1b4c51a96a67b52c38fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" is_pre_election: true
I20260811 04:36:30.180569 4567 raft_consensus.cc:2454] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907b07892c1b4c51a96a67b52c38fba7 in term 0.
I20260811 04:36:30.180588 4725 raft_consensus.cc:2454] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 907b07892c1b4c51a96a67b52c38fba7 in term 0.
I20260811 04:36:30.180970 4331 leader_election.cc:302] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495e1e0e98c46d3a68fafabe1f6b3b1, 907b07892c1b4c51a96a67b52c38fba7; no voters:
I20260811 04:36:30.181259 4814 raft_consensus.cc:2781] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:30.181317 4814 raft_consensus.cc:482] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:30.181345 4814 raft_consensus.cc:3037] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:30.182441 4814 raft_consensus.cc:504] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:30.182844 4814 leader_election.cc:288] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [CANDIDATE]: Term 1 election: Requested vote from peers e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151), 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025)
I20260811 04:36:30.183248 4567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467d39db9dc34b60b7ef6aa4b09e9d00" candidate_uuid: "907b07892c1b4c51a96a67b52c38fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e745992bb906476b9f178e12c1c6880c"
I20260811 04:36:30.183251 4725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "467d39db9dc34b60b7ef6aa4b09e9d00" candidate_uuid: "907b07892c1b4c51a96a67b52c38fba7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1"
I20260811 04:36:30.183385 4567 raft_consensus.cc:3037] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:30.183394 4725 raft_consensus.cc:3037] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:30.184324 4725 raft_consensus.cc:2454] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907b07892c1b4c51a96a67b52c38fba7 in term 1.
I20260811 04:36:30.184435 4567 raft_consensus.cc:2454] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 907b07892c1b4c51a96a67b52c38fba7 in term 1.
I20260811 04:36:30.184677 4331 leader_election.cc:302] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2495e1e0e98c46d3a68fafabe1f6b3b1, 907b07892c1b4c51a96a67b52c38fba7; no voters:
I20260811 04:36:30.184854 4814 raft_consensus.cc:2781] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:30.185088 4814 raft_consensus.cc:686] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 LEADER]: Becoming Leader. State: Replica: 907b07892c1b4c51a96a67b52c38fba7, State: Running, Role: LEADER
I20260811 04:36:30.185403 4814 consensus_queue.cc:237] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } }
I20260811 04:36:30.187288 4855 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: TxnStatusTablet state changed. Reason: New leader 907b07892c1b4c51a96a67b52c38fba7. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.187397 4855 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:30.187705 4858 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 04:36:30.187924 4858 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 04:36:30.188249 4224 catalog_manager.cc:5495] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129). New cstate: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:30.249992 4815 consensus_queue.cc:1035] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:30.256306 4815 consensus_queue.cc:1035] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [LEADER]: Connected to new peer: Peer: permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:30.269457 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.265392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37899 (local address 127.0.0.1:47736)
0811 04:36:30.265541 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:30.265559 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:30.265820 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:30.266176 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:30.266179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:30.266207 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:30.266322 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.266327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.267055 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.267060 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.267760 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.267769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.268273 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.268276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.268375 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.268420 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.269151 (+ 731us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.269168 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 04:36:30.269248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 04:36:30.269346 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 04:36:30.269457 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.265738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47736 (local address 127.2.238.129:37899)
0811 04:36:30.265903 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.265908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.265921 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:30.265993 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.265995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.266058 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.266161 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:30.266430 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.266942 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.267879 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.268152 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.268261 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.268612 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.269047 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.269181 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.269256 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.269322 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 04:36:30.272435 4725 raft_consensus.cc:1261] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Refusing update from remote peer 907b07892c1b4c51a96a67b52c38fba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:30.272468 4567 raft_consensus.cc:1261] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Refusing update from remote peer 907b07892c1b4c51a96a67b52c38fba7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:30.272894 4855 consensus_queue.cc:1035] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:30.273080 4814 consensus_queue.cc:1035] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:30.274670 4817 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: TxnStatusTablet state changed. Reason: New leader 907b07892c1b4c51a96a67b52c38fba7. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.274801 4817 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:30.275439 4815 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: TxnStatusTablet state changed. Reason: New leader 907b07892c1b4c51a96a67b52c38fba7. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.275544 4815 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:30.276813 4855 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.276913 4855 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:30.277027 4864 mvcc.cc:205] Tried to move back new op lower bound from 7317188568150949888 to 7317188567802372096. Current Snapshot: MvccSnapshot[applied={T|T < 7317188568150949888}]
I20260811 04:36:30.278214 4814 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.278369 4814 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 907b07892c1b4c51a96a67b52c38fba7: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:30.278769 4815 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.278872 4815 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P e745992bb906476b9f178e12c1c6880c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:30.278851 4862 mvcc.cc:205] Tried to move back new op lower bound from 7317188568150949888 to 7317188567802372096. Current Snapshot: MvccSnapshot[applied={T|T < 7317188568150949888}]
I20260811 04:36:30.279258 4861 mvcc.cc:205] Tried to move back new op lower bound from 7317188568150949888 to 7317188567802372096. Current Snapshot: MvccSnapshot[applied={T|T < 7317188568150949888}]
I20260811 04:36:30.279309 4817 tablet_replica.cc:439] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "907b07892c1b4c51a96a67b52c38fba7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "907b07892c1b4c51a96a67b52c38fba7" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37899 } } peers { permanent_uuid: "e745992bb906476b9f178e12c1c6880c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40151 } } peers { permanent_uuid: "2495e1e0e98c46d3a68fafabe1f6b3b1" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40025 } } }
I20260811 04:36:30.279428 4817 tablet_replica.cc:442] T 467d39db9dc34b60b7ef6aa4b09e9d00 P 2495e1e0e98c46d3a68fafabe1f6b3b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:30.287144 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.283152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52664 (local address 127.2.238.130:40151)
0811 04:36:30.283266 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.283271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.283545 (+ 274us) server_negotiation.cc:407] Connection header received
0811 04:36:30.283694 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.283697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.283759 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.283861 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:30.284090 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.284805 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.285588 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.285963 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.286074 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.286295 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.286798 (+ 503us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.286929 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.286966 (+ 37us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.287023 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:36:30.298164 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.294397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53509 (local address 127.2.238.129:37899)
0811 04:36:30.294537 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.294542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.294664 (+ 122us) server_negotiation.cc:407] Connection header received
0811 04:36:30.294818 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.294821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.294873 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.294950 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:30.295173 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.295650 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.297052 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.297266 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.297383 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.297674 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.297751 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.297863 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.297949 (+ 86us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.297996 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 04:36:30.298199 4876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.294297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37899 (local address 127.2.238.130:53509)
0811 04:36:30.294586 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:30.294599 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:30.294720 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:30.294960 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:30.294962 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:30.294969 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:30.295070 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.295075 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.295765 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.295770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.296921 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.296931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.297403 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.297407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.297484 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.297522 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.297870 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.297878 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:36:30.297940 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 04:36:30.298013 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":95,"threads_started":1}
I20260811 04:36:30.327433 4853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.323494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:53665 (local address 127.2.238.130:40151)
0811 04:36:30.323655 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.323660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.323680 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:36:30.323732 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.323735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.323790 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.323900 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:30.324240 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.325015 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.326182 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.326483 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.326599 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.326955 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.327047 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.327133 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.327274 (+ 141us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.327324 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 04:36:30.327723 4880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.323142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40151 (local address 127.2.238.129:53665)
0811 04:36:30.323506 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:30.323524 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:30.323634 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:30.323989 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:30.323992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:30.323999 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:30.324123 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.324128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.325144 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.325146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.326072 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:30.326087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.326631 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.326636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:30.326746 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.326793 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:30.327180 (+ 387us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:30.327187 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 04:36:30.327255 (+ 68us) client_negotiation.cc:240] Negotiation successful
0811 04:36:30.327343 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":160,"threads_started":1}
I20260811 04:36:30.352196 4225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:30.352376 4225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:30.354890 4225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41b65340a2d3427bb2c1b9963c047a28: Sending DeleteTablet for 3 replicas of tablet 9adce377d4e0462d880b88c399749262
I20260811 04:36:30.355556 4381 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adce377d4e0462d880b88c399749262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:30 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:47726
I20260811 04:36:30.355768 4703 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adce377d4e0462d880b88c399749262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:30 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:35988
I20260811 04:36:30.355962 4883 tablet_replica.cc:330] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: stopping tablet replica
I20260811 04:36:30.356084 4883 raft_consensus.cc:2229] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:30.356213 4883 raft_consensus.cc:2258] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:30.356452 4545 tablet_service.cc:1514] Processing DeleteTablet for tablet 9adce377d4e0462d880b88c399749262 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:30 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52660
I20260811 04:36:30.356907 4883 ts_tablet_manager.cc:1905] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:30.358074 4884 tablet_replica.cc:330] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: stopping tablet replica
I20260811 04:36:30.358541 4885 tablet_replica.cc:330] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: stopping tablet replica
I20260811 04:36:30.359228 4885 raft_consensus.cc:2229] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:30.359817 4884 raft_consensus.cc:2229] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:30.359951 4883 ts_tablet_manager.cc:1918] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:30.360052 4883 log.cc:1196] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/9adce377d4e0462d880b88c399749262
I20260811 04:36:30.360224 4884 raft_consensus.cc:2258] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:30.360386 4883 ts_tablet_manager.cc:1939] T 9adce377d4e0462d880b88c399749262 P 907b07892c1b4c51a96a67b52c38fba7: Deleting consensus metadata
I20260811 04:36:30.360616 4885 raft_consensus.cc:2258] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:30.361192 4212 catalog_manager.cc:4841] TS 907b07892c1b4c51a96a67b52c38fba7 (127.2.238.129:37899): tablet 9adce377d4e0462d880b88c399749262 (table test-table [id=2691b3d203f3463abe670cb9b28d0d56]) successfully deleted
I20260811 04:36:30.361946 4885 ts_tablet_manager.cc:1905] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:30.362068 4884 ts_tablet_manager.cc:1905] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:30.365288 4885 ts_tablet_manager.cc:1918] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:30.365346 4884 ts_tablet_manager.cc:1918] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:30.365435 4884 log.cc:1196] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/9adce377d4e0462d880b88c399749262
I20260811 04:36:30.365458 4885 log.cc:1196] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/9adce377d4e0462d880b88c399749262
I20260811 04:36:30.365854 4884 ts_tablet_manager.cc:1939] T 9adce377d4e0462d880b88c399749262 P 2495e1e0e98c46d3a68fafabe1f6b3b1: Deleting consensus metadata
I20260811 04:36:30.365872 4885 ts_tablet_manager.cc:1939] T 9adce377d4e0462d880b88c399749262 P e745992bb906476b9f178e12c1c6880c: Deleting consensus metadata
I20260811 04:36:30.366670 4212 catalog_manager.cc:4841] TS e745992bb906476b9f178e12c1c6880c (127.2.238.130:40151): tablet 9adce377d4e0462d880b88c399749262 (table test-table [id=2691b3d203f3463abe670cb9b28d0d56]) successfully deleted
I20260811 04:36:30.367000 4212 catalog_manager.cc:4841] TS 2495e1e0e98c46d3a68fafabe1f6b3b1 (127.2.238.131:40025): tablet 9adce377d4e0462d880b88c399749262 (table test-table [id=2691b3d203f3463abe670cb9b28d0d56]) successfully deleted
Aug 11 04:36:30 dist-test-slave-0kls krb5kdc[4176](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422989, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:30.371385 4802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.358979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.2.238.129:37899)
0811 04:36:30.359101 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.359106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.360030 (+ 924us) server_negotiation.cc:407] Connection header received
0811 04:36:30.360243 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.360249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.360308 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.360365 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:30.361369 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.361967 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.363018 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.363315 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.363389 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.367779 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:30.367790 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:30.367807 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:30.367837 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:30.369633 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:30.370089 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:30.370095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:30.370097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:30.370193 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:30.370460 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:30.370464 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:30.370465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:30.370734 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:30.370942 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.371098 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.371225 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260811 04:36:30.371807 4450 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47748
I20260811 04:36:30.381160 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:30.373580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.2.238.190:34837)
0811 04:36:30.373716 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:30.373719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:30.373757 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:36:30.373926 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:30.373927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:30.373966 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:30.374025 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:30.374720 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.375241 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.376349 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:30.376554 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:30.376629 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:30.378155 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:30.378158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:30.378159 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:30.378183 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:30.379572 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:30.380040 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:30.380043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:30.380044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:30.380117 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:30.380387 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:30.380389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:30.380390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:30.380552 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:30.380654 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:30.380884 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 04:36:30.380991 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 04:36:30.381588 4225 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33976
I20260811 04:36:30.382985 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 4309
I20260811 04:36:30.395926 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 4468
I20260811 04:36:30.404279 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 4632
I20260811 04:36:30.413045 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 4191
2026-08-11T04:36:30Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3110 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:30 dist-test-slave-0kls krb5kdc[4892](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:30 dist-test-slave-0kls krb5kdc[4892](info): set up 2 sockets
Aug 11 04:36:30 dist-test-slave-0kls krb5kdc[4892](info): commencing operation
krb5kdc: starting...
W20260811 04:36:31.490314 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:31 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:31.620819 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46799
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:31.736148 4906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:31.736471 4906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:31.736559 4906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:31.740692 4906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:31.740793 4906 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 04:36:31.740825 4906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:31.740870 4906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:31.740907 4906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:31.740942 4906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:31.746496 4906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46799
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:31.747706 4906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:31.748818 4906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:31.757608 4912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:31.757731 4906 server_base.cc:1034] running on GCE node
W20260811 04:36:31.757603 4915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:31.757608 4913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:31.758581 4906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:31.759739 4906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:31.761027 4906 hybrid_clock.cc:648] HybridClock initialized: now 1786422991761012 us; error 32 us; skew 500 ppm
Aug 11 04:36:31 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422991, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:31.764207 4906 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:31.765268 4906 webserver.cc:460] Webserver started at http://127.2.238.190:46303/ using document root <none> and password file <none>
I20260811 04:36:31.765924 4906 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:31.766004 4906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:31.766208 4906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:31.768535 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "26bb60cbaf5e40faa095c237192f2c29"
format_stamp: "Formatted at 2026-08-11 04:36:31 on dist-test-slave-0kls"
server_key: "729b20f1305058bce658f6128d8214c1"
server_key_iv: "2d1d14df750c4b234ab1f88cd93afdb5"
server_key_version: "encryptionkey@0"
I20260811 04:36:31.769126 4906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "26bb60cbaf5e40faa095c237192f2c29"
format_stamp: "Formatted at 2026-08-11 04:36:31 on dist-test-slave-0kls"
server_key: "729b20f1305058bce658f6128d8214c1"
server_key_iv: "2d1d14df750c4b234ab1f88cd93afdb5"
server_key_version: "encryptionkey@0"
I20260811 04:36:31.773211 4906 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:31.775961 4925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:31.777400 4906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:31.777518 4906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "26bb60cbaf5e40faa095c237192f2c29"
format_stamp: "Formatted at 2026-08-11 04:36:31 on dist-test-slave-0kls"
server_key: "729b20f1305058bce658f6128d8214c1"
server_key_iv: "2d1d14df750c4b234ab1f88cd93afdb5"
server_key_version: "encryptionkey@0"
I20260811 04:36:31.777619 4906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:31.787312 4906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:31.790709 4906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:31.790877 4906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:31.799515 4906 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:46799
I20260811 04:36:31.799542 4995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:46799 every 8 connection(s)
I20260811 04:36:31.800529 4906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:31.803476 4996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:31.808959 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 4906
I20260811 04:36:31.809084 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:31.809356 3002 external_mini_cluster.cc:1428] Setting key 58b10adb1a7a7296cc72dc38a7a83eeb
I20260811 04:36:31.809669 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Bootstrap starting.
I20260811 04:36:31.812486 4996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:31.813280 4996 log.cc:824] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:31.815281 4996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: No bootstrap required, opened a new log
Aug 11 04:36:31 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:31.818658 4996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } }
I20260811 04:36:31.818928 4996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:31.819020 4996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26bb60cbaf5e40faa095c237192f2c29, State: Initialized, Role: FOLLOWER
I20260811 04:36:31.819583 4996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } }
I20260811 04:36:31.819766 4996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:31.819911 4996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:31.820016 4996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:31.820955 4996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } }
I20260811 04:36:31.821326 4996 leader_election.cc:302] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26bb60cbaf5e40faa095c237192f2c29; no voters:
I20260811 04:36:31.821696 4996 leader_election.cc:288] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:31.821849 5001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:31.822104 5001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [term 1 LEADER]: Becoming Leader. State: Replica: 26bb60cbaf5e40faa095c237192f2c29, State: Running, Role: LEADER
I20260811 04:36:31.822464 5001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } }
I20260811 04:36:31.823088 4996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:31.824410 5003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26bb60cbaf5e40faa095c237192f2c29. Latest consensus state: current_term: 1 leader_uuid: "26bb60cbaf5e40faa095c237192f2c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } } }
I20260811 04:36:31.824435 5002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26bb60cbaf5e40faa095c237192f2c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26bb60cbaf5e40faa095c237192f2c29" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46799 } } }
I20260811 04:36:31.824550 5003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:31.824553 5002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:31.824918 5010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:31.825349 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:31.810925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.2.238.190:46799)
0811 04:36:31.811373 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:31.811390 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:31.811426 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:36:31.812062 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:31.812079 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:31.812420 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:31.812784 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:31.813671 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:31.814766 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:31.815739 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:31.816216 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:31.816880 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:31.819109 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:31.819118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:31.819138 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:31.819178 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:31.821128 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:31.821637 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:31.821643 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:31.821650 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:31.821893 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:31.822269 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:31.822272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:31.822274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:31.822825 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:31.823751 (+ 926us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:31.823912 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 04:36:31.824245 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":142,"threads_started":1}
I20260811 04:36:31.828210 5010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:31.834306 5010 catalog_manager.cc:1347] Generated new cluster ID: 4d6b4166629a4af291791909b01a6f34
I20260811 04:36:31.834414 5010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:31.874032 5010 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:31.875036 5010 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:31.881613 5010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Generated new TSK 0
I20260811 04:36:31.882335 5010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:31.920982 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46799
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:32.041999 5022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:32.042294 5022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:32.042346 5022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:32.046715 5022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:32.046799 5022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:32.046844 5022 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 04:36:32.046860 5022 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:32.046926 5022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:32.052601 5022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:32.053846 5022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:32.054936 5022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:32.065094 5031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.065094 5028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.065131 5027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:32.065668 5022 server_base.cc:1034] running on GCE node
I20260811 04:36:32.066102 5022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:32.066756 5022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:32.067920 5022 hybrid_clock.cc:648] HybridClock initialized: now 1786422992067890 us; error 42 us; skew 500 ppm
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:32.070845 5022 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:32.071872 5022 webserver.cc:460] Webserver started at http://127.2.238.129:42459/ using document root <none> and password file <none>
I20260811 04:36:32.072474 5022 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:32.072537 5022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:32.072788 5022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:32.075027 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "77782ec70b62436587ca90ce3a578716"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "be2dc8c2e0e1d2017055e06cea7c10ad"
server_key_iv: "5e1c42decc1fd1e2cd16357e74370fcf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.075649 5022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "77782ec70b62436587ca90ce3a578716"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "be2dc8c2e0e1d2017055e06cea7c10ad"
server_key_iv: "5e1c42decc1fd1e2cd16357e74370fcf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.079742 5022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:32.082671 5040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.084190 5022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:32.084340 5022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "77782ec70b62436587ca90ce3a578716"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "be2dc8c2e0e1d2017055e06cea7c10ad"
server_key_iv: "5e1c42decc1fd1e2cd16357e74370fcf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.084465 5022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:32.110095 5022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:32.113410 5022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:32.113636 5022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:32.115298 5022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:32.115353 5022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.115396 5022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:32.115458 5022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:32.127012 5022 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:43071
I20260811 04:36:32.127060 5169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:43071 every 8 connection(s)
I20260811 04:36:32.127955 5022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:32.129451 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5022
I20260811 04:36:32.129585 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:32.129992 3002 external_mini_cluster.cc:1428] Setting key 9407e2e8cacbf82b5a7fca46c0563a87
I20260811 04:36:32.131597 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.116432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:36853 (local address 127.2.238.190:46799)
0811 04:36:32.116570 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.116574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.117712 (+ 1138us) server_negotiation.cc:407] Connection header received
0811 04:36:32.118967 (+ 1255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.118970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.119038 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.119137 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.120933 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.121871 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.123895 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.124214 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.124307 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.127370 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.127374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.127377 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.127414 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.129089 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.129720 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.129724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.129726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.129864 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.130355 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.130360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.130365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.130578 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.130701 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.131270 (+ 569us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.131424 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:32.132431 5051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.117009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.129:36853)
0811 04:36:32.117547 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.117591 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.118688 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.119369 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.119382 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.119918 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.120725 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.120752 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.122026 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.122032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.123726 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.123740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.124398 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.124404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.124637 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.125418 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.125439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.127168 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.129216 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.129222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.129247 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.129620 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.129979 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.129983 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.129985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.130234 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.130729 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.130734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.130980 (+ 246us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.131214 (+ 234us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.131557 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":154,"threads_started":1}
I20260811 04:36:32.134243 5170 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46799
I20260811 04:36:32.134413 5170 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:32.135042 5170 heartbeater.cc:511] Master 127.2.238.190:46799 requested a full tablet report, sending...
I20260811 04:36:32.136694 4944 ts_manager.cc:194] Registered new tserver with Master: 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071)
I20260811 04:36:32.138370 4944 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:36853
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:32.145170 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.137150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.129:38131)
0811 04:36:32.137503 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.137518 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.137622 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.137939 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.137942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.138181 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.138312 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.138320 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.139490 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.139493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.140336 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.140346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.141044 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.141047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.141123 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.141694 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.141706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.142101 (+ 395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.143860 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.143863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.143865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.144062 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.144373 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.144375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.144376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.144452 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.144746 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.144748 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.144827 (+ 79us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.144910 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.145024 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":100,"threads_started":1}
I20260811 04:36:32.145282 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.137219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:38131 (local address 127.2.238.190:46799)
0811 04:36:32.137366 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.137373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.137598 (+ 225us) server_negotiation.cc:407] Connection header received
0811 04:36:32.137717 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.137723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.137809 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.137912 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.138445 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.139340 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.140477 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.140899 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.141019 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.142262 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.142266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.142270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.142301 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.143757 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.144163 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.144167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.144170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.144284 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.144511 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.144513 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.144514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.144671 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.144748 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.144940 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.145081 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:32.163244 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46799
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:32.282126 5180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:32.282418 5180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:32.282469 5180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:32.286579 5180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:32.286645 5180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:32.286688 5180 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 04:36:32.286702 5180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:32.286738 5180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:32.292342 5180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:32.293494 5180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:32.294667 5180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:32.304710 5186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.304728 5191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.304728 5187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:32.304999 5180 server_base.cc:1034] running on GCE node
I20260811 04:36:32.305428 5180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:32.306214 5180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:32.307403 5180 hybrid_clock.cc:648] HybridClock initialized: now 1786422992307364 us; error 56 us; skew 500 ppm
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:32.310572 5180 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:32.311614 5180 webserver.cc:460] Webserver started at http://127.2.238.130:33713/ using document root <none> and password file <none>
I20260811 04:36:32.312248 5180 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:32.312350 5180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:32.312599 5180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:32.314934 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "14e0f678e82e4b30a8975ac0b228805b"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "073aded005a0a39e354d3146e7cf6dfb"
server_key_iv: "5692dc18faafea4bba75614dbbb0b2cf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.315627 5180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "14e0f678e82e4b30a8975ac0b228805b"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "073aded005a0a39e354d3146e7cf6dfb"
server_key_iv: "5692dc18faafea4bba75614dbbb0b2cf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.319970 5180 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:32.322904 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.324401 5180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260811 04:36:32.324571 5180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "14e0f678e82e4b30a8975ac0b228805b"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "073aded005a0a39e354d3146e7cf6dfb"
server_key_iv: "5692dc18faafea4bba75614dbbb0b2cf"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.324680 5180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:32.354552 5180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:32.357573 5180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:32.357824 5180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:32.359361 5180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:32.359438 5180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.359508 5180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:32.359558 5180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:32.370669 5180 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43279
I20260811 04:36:32.370957 5331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43279 every 8 connection(s)
I20260811 04:36:32.371862 5180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:32.373963 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.360259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:43699 (local address 127.2.238.190:46799)
0811 04:36:32.360438 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.360445 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.361278 (+ 833us) server_negotiation.cc:407] Connection header received
0811 04:36:32.362406 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.362410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.362470 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.362536 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.364452 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.365328 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.366797 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.367037 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.367174 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.369868 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.369873 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.369879 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.369917 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.371622 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.372210 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.372214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.372216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.372321 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.372771 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.372775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.372776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.372976 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.373093 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.373599 (+ 506us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.373752 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:36:32.374732 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.360657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.130:43699)
0811 04:36:32.361104 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.361155 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.362147 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.363011 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.363023 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.363532 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.364263 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.364284 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.365482 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.365489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.366667 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.366677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.367231 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.367235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.367411 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.368017 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.368033 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.369652 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.371764 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.371769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.371780 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.372084 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.372414 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.372419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.372421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.372672 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.373092 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.373098 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.373352 (+ 254us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.373554 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.373927 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":124,"threads_started":1}
I20260811 04:36:32.375975 5332 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46799
I20260811 04:36:32.376137 5332 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:32.376673 5332 heartbeater.cc:511] Master 127.2.238.190:46799 requested a full tablet report, sending...
I20260811 04:36:32.377684 4942 ts_manager.cc:194] Registered new tserver with Master: 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279)
I20260811 04:36:32.378441 4942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:43699
I20260811 04:36:32.381402 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5180
I20260811 04:36:32.381526 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:32.381958 3002 external_mini_cluster.cc:1428] Setting key 2d10f4fa2f8a89b41f671b6ccde547d1
I20260811 04:36:32.385767 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.378363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:55249 (local address 127.2.238.190:46799)
0811 04:36:32.378498 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.378501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.378631 (+ 130us) server_negotiation.cc:407] Connection header received
0811 04:36:32.378761 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.378764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.378812 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.378890 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.379327 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.379863 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.380774 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.381101 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.381184 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.382510 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.382514 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.382516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.382544 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.384030 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.384533 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.384536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.384537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.384610 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.384906 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.384908 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.384909 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.385106 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.385182 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.385498 (+ 316us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.385636 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:36:32.385998 5338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.378244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.130:55249)
0811 04:36:32.378550 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.378566 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.378663 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.378910 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.378913 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.379101 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.379211 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.379217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.379991 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.379994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.380591 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.380600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.381218 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.381222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.381321 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.381889 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.381903 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.382361 (+ 458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.384154 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.384160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.384165 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.384432 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.384701 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.384707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.384712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.384816 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.385201 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.385207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.385322 (+ 115us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.385449 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.385629 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:32.412554 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46799
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:32.530643 5342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:32.530973 5342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:32.531080 5342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:32.535209 5342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:32.535300 5342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:32.535375 5342 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260811 04:36:32.535419 5342 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:36:32.535491 5342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:32.540946 5342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:46799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:32.542165 5342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:32.543270 5342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:32.553611 5347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.553611 5350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:32.553678 5348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:32.553879 5342 server_base.cc:1034] running on GCE node
I20260811 04:36:32.554314 5342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:32.555004 5342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:32.556185 5342 hybrid_clock.cc:648] HybridClock initialized: now 1786422992556170 us; error 24 us; skew 500 ppm
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:32.559254 5342 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:32.560447 5342 webserver.cc:460] Webserver started at http://127.2.238.131:33731/ using document root <none> and password file <none>
I20260811 04:36:32.561053 5342 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:32.561149 5342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:32.561395 5342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:32.563592 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "539d933aed37bd5554014dcb88e1ba25"
server_key_iv: "9f35c4e567cf5776bb4d749525432869"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.564210 5342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "539d933aed37bd5554014dcb88e1ba25"
server_key_iv: "9f35c4e567cf5776bb4d749525432869"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.568138 5342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:32.570678 5359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.571959 5342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:32.572103 5342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
format_stamp: "Formatted at 2026-08-11 04:36:32 on dist-test-slave-0kls"
server_key: "539d933aed37bd5554014dcb88e1ba25"
server_key_iv: "9f35c4e567cf5776bb4d749525432869"
server_key_version: "encryptionkey@0"
I20260811 04:36:32.572220 5342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:32.592067 5342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:32.595582 5342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:32.595822 5342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:32.597316 5342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:32.597390 5342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:32.597462 5342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:32.597514 5342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:32.609429 5342 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:41045
I20260811 04:36:32.609493 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:41045 every 8 connection(s)
I20260811 04:36:32.610590 5342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:32.611994 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.598254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:49557 (local address 127.2.238.190:46799)
0811 04:36:32.598416 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.598421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.599511 (+ 1090us) server_negotiation.cc:407] Connection header received
0811 04:36:32.600578 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.600583 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.600652 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.600757 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.602437 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.603056 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.604631 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.604988 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.605121 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.607854 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.607858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.607860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.607889 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.609569 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.610138 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.610141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.610143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.610262 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.610621 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.610627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.610630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.610888 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.611006 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.611767 (+ 761us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.611871 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:36:32.612771 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.598849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.131:49557)
0811 04:36:32.599362 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.599405 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.600324 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.601068 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.601080 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.601636 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.602287 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.602306 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.603194 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.603198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.604503 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.604513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.605179 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.605184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.605389 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.606052 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.606079 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.607681 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.609675 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.609680 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.609691 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.610031 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.610356 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.610358 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.610359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.610504 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.611023 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.611029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.611319 (+ 290us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.611647 (+ 328us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.611904 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":144,"threads_started":1}
I20260811 04:36:32.614126 5484 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46799
I20260811 04:36:32.614377 5484 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:32.614960 5484 heartbeater.cc:511] Master 127.2.238.190:46799 requested a full tablet report, sending...
I20260811 04:36:32.616050 4942 ts_manager.cc:194] Registered new tserver with Master: e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045)
I20260811 04:36:32.616853 4942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:49557
I20260811 04:36:32.620239 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5342
I20260811 04:36:32.620384 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:32.620783 3002 external_mini_cluster.cc:1428] Setting key 79b7b910c71d977f7e2b67e1a2cb900f
I20260811 04:36:32.623304 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:36:32.624433 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.616633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40057 (local address 127.2.238.190:46799)
0811 04:36:32.616754 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.616757 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.616971 (+ 214us) server_negotiation.cc:407] Connection header received
0811 04:36:32.617119 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.617122 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.617168 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.617264 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.617827 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.618330 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.619266 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.619464 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.619540 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.620850 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.620855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.620858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.620888 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.622581 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.623096 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.623099 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.623101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.623216 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.623519 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.623521 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.623522 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.623689 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.623792 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.624171 (+ 379us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.624309 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:32.624544 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.616553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.2.238.131:40057)
0811 04:36:32.616885 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.616897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.617018 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.617280 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.617283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.617545 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.617659 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.617667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.618470 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.618474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.619161 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.619170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.619593 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.619597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.619678 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.620185 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.620204 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.620686 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.622700 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.622702 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.622704 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.622999 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.623316 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.623322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.623327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.623436 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.623812 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.623815 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.623928 (+ 113us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.624043 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.624276 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:32.650615 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.642788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40326 (local address 127.2.238.190:46799)
0811 04:36:32.642923 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.642926 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.642951 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:36:32.642997 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.642999 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.643041 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.643117 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.643794 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.644292 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.645060 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.645259 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.645331 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.647816 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.647819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.647821 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.647846 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.649171 (+ 1325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.649542 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.649544 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.649545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.649618 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.649902 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.649904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.649905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.650086 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.650187 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.650396 (+ 209us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.650492 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:36:32.653541 4942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:32.656772 4942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:32.672673 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.665920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.0.0.1:59278)
0811 04:36:32.666970 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.666984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.667082 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.667765 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.667767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.667781 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.667852 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.667859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.669379 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.669382 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.670944 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.670953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.671982 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.671986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.672163 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.672207 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.672315 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.672500 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":330,"thread_start_us":123,"threads_started":1}
I20260811 04:36:32.672909 5505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.666551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:43071 (local address 127.0.0.1:45230)
0811 04:36:32.667351 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.667364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.667449 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.667923 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.667927 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.667946 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.668046 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.668056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.669030 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.669034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.670626 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.670640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.672516 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.672520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.672611 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.672635 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.672701 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.672783 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":128,"thread_start_us":111,"threads_started":1}
I20260811 04:36:32.673218 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.666995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45230 (local address 127.2.238.129:43071)
0811 04:36:32.667457 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.667464 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.667493 (+ 29us) server_negotiation.cc:407] Connection header received
0811 04:36:32.667577 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.667582 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.667775 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.667948 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.668175 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.668910 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.670786 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.672298 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.672521 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.672815 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.672882 (+ 67us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.673002 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":73,"threads_started":1}
I20260811 04:36:32.673379 5504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.665813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43279 (local address 127.0.0.1:55998)
0811 04:36:32.666227 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.666289 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.666416 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.667642 (+ 1226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.667648 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.667678 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.667852 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.667859 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.669370 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.669373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.670828 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.670840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.671982 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.671986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.673073 (+ 1087us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.673097 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.673179 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.673252 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1003,"thread_start_us":102,"threads_started":1}
I20260811 04:36:32.674304 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.665927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55998 (local address 127.2.238.130:43279)
0811 04:36:32.666750 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.666764 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.666788 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:36:32.666871 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.666876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.667099 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.667624 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.667991 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.669203 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.670978 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.671829 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.672119 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.672305 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.673218 (+ 913us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.673324 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":114,"threads_started":1}
I20260811 04:36:32.674321 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.666553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.2.238.131:41045)
0811 04:36:32.667353 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.667363 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.667381 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:36:32.667440 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.667444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.667614 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.668006 (+ 392us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.668168 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.669260 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.671064 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.671829 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.673632 (+ 1803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.673883 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.674010 (+ 127us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.674094 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":113,"threads_started":1}
I20260811 04:36:32.676606 5098 tablet_service.cc:1467] Processing CreateTablet for tablet f51d698569f849a09887c670a9392120 (DEFAULT_TABLE table=test-table [id=8e61e1d3079541c0872aa5733ed5a455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:32.677219 5414 tablet_service.cc:1467] Processing CreateTablet for tablet f51d698569f849a09887c670a9392120 (DEFAULT_TABLE table=test-table [id=8e61e1d3079541c0872aa5733ed5a455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:32.677198 5256 tablet_service.cc:1467] Processing CreateTablet for tablet f51d698569f849a09887c670a9392120 (DEFAULT_TABLE table=test-table [id=8e61e1d3079541c0872aa5733ed5a455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:32.677752 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51d698569f849a09887c670a9392120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.678308 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51d698569f849a09887c670a9392120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.678308 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f51d698569f849a09887c670a9392120. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.684238 5510 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Bootstrap starting.
I20260811 04:36:32.686296 5512 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Bootstrap starting.
I20260811 04:36:32.687213 5511 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Bootstrap starting.
I20260811 04:36:32.687335 5510 tablet_bootstrap.cc:654] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.688304 5510 log.cc:824] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:32.688869 5512 tablet_bootstrap.cc:654] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.689173 5511 tablet_bootstrap.cc:654] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.689710 5512 log.cc:824] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:32.689921 5511 log.cc:824] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:32.690732 5510 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: No bootstrap required, opened a new log
I20260811 04:36:32.690960 5510 ts_tablet_manager.cc:1397] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:36:32.691753 5511 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: No bootstrap required, opened a new log
I20260811 04:36:32.691951 5511 ts_tablet_manager.cc:1397] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260811 04:36:32.692014 5512 tablet_bootstrap.cc:492] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: No bootstrap required, opened a new log
I20260811 04:36:32.692189 5512 ts_tablet_manager.cc:1397] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20260811 04:36:32.694931 5511 raft_consensus.cc:348] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.695178 5511 raft_consensus.cc:374] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.695231 5511 raft_consensus.cc:729] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.695362 5510 raft_consensus.cc:348] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.695617 5510 raft_consensus.cc:374] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.695681 5510 raft_consensus.cc:729] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.695744 5511 consensus_queue.cc:260] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.695959 5512 raft_consensus.cc:348] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.696089 5510 consensus_queue.cc:260] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.696198 5512 raft_consensus.cc:374] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.696249 5512 raft_consensus.cc:729] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.696624 5484 heartbeater.cc:503] Master 127.2.238.190:46799 was elected leader, sending a full tablet report...
I20260811 04:36:32.696678 5512 consensus_queue.cc:260] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.696856 5511 ts_tablet_manager.cc:1428] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260811 04:36:32.697563 5512 ts_tablet_manager.cc:1428] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260811 04:36:32.697563 5510 ts_tablet_manager.cc:1428] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 04:36:32.697813 5332 heartbeater.cc:503] Master 127.2.238.190:46799 was elected leader, sending a full tablet report...
I20260811 04:36:32.697937 5170 heartbeater.cc:503] Master 127.2.238.190:46799 was elected leader, sending a full tablet report...
I20260811 04:36:32.847399 5519 raft_consensus.cc:482] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:32.847653 5519 raft_consensus.cc:504] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.848929 5519 leader_election.cc:288] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045), 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071)
I20260811 04:36:32.853432 5208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.849159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.2.238.130:42677)
0811 04:36:32.849309 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.849328 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.849445 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.849740 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.849745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.849772 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.849973 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.849982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.851036 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.851039 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.852351 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.852363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.853092 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.853094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.853178 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.853199 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.853265 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.853321 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":1}
I20260811 04:36:32.853518 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.849309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:42677 (local address 127.2.238.131:41045)
0811 04:36:32.849473 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.849477 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.849495 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:36:32.849557 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.849563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.849626 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.849697 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.850103 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.850925 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.852492 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.852975 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.853094 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.853150 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.853309 (+ 159us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.853383 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:36:32.854130 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.849313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:43071 (local address 127.2.238.130:53305)
0811 04:36:32.849740 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.849753 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.849881 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.850151 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.850153 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.850168 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.850257 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.850261 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.851207 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.851210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.852729 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.852746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.853732 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.853738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.853853 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.853879 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.853961 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.854010 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":25,"thread_start_us":116,"threads_started":1}
I20260811 04:36:32.854199 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.849482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53305 (local address 127.2.238.129:43071)
0811 04:36:32.849601 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.849606 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.849889 (+ 283us) server_negotiation.cc:407] Connection header received
0811 04:36:32.849976 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.849978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.850020 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.850133 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.850376 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.851108 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.852858 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.853462 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.853581 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.853634 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.854008 (+ 374us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.854091 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 04:36:32.854226 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51d698569f849a09887c670a9392120" candidate_uuid: "14e0f678e82e4b30a8975ac0b228805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" is_pre_election: true
I20260811 04:36:32.854554 5435 raft_consensus.cc:2454] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e0f678e82e4b30a8975ac0b228805b in term 0.
I20260811 04:36:32.854851 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51d698569f849a09887c670a9392120" candidate_uuid: "14e0f678e82e4b30a8975ac0b228805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716" is_pre_election: true
I20260811 04:36:32.855116 5200 leader_election.cc:302] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e0f678e82e4b30a8975ac0b228805b, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:32.855134 5120 raft_consensus.cc:2454] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14e0f678e82e4b30a8975ac0b228805b in term 0.
I20260811 04:36:32.855429 5519 raft_consensus.cc:2781] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:32.855530 5519 raft_consensus.cc:482] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:32.855607 5519 raft_consensus.cc:3037] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.856668 5519 raft_consensus.cc:504] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
I20260811 04:36:32.857036 5519 leader_election.cc:288] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [CANDIDATE]: Term 1 election: Requested vote from peers e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045), 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071)
I20260811 04:36:32.857395 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51d698569f849a09887c670a9392120" candidate_uuid: "14e0f678e82e4b30a8975ac0b228805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
I20260811 04:36:32.857527 5435 raft_consensus.cc:3037] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.857560 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f51d698569f849a09887c670a9392120" candidate_uuid: "14e0f678e82e4b30a8975ac0b228805b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716"
I20260811 04:36:32.857728 5120 raft_consensus.cc:3037] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.858664 5435 raft_consensus.cc:2454] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e0f678e82e4b30a8975ac0b228805b in term 1.
I20260811 04:36:32.858983 5200 leader_election.cc:302] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e0f678e82e4b30a8975ac0b228805b, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:32.859112 5120 raft_consensus.cc:2454] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14e0f678e82e4b30a8975ac0b228805b in term 1.
I20260811 04:36:32.859154 5519 raft_consensus.cc:2781] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:32.859411 5519 raft_consensus.cc:686] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 1 LEADER]: Becoming Leader. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Running, Role: LEADER
I20260811 04:36:32.859762 5519 consensus_queue.cc:237] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } }
W20260811 04:36:32.861617 5485 tablet.cc:2369] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:36:32.863054 4941 catalog_manager.cc:5495] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130). New cstate: current_term: 1 leader_uuid: "14e0f678e82e4b30a8975ac0b228805b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } health_report { overall_health: UNKNOWN } } }
W20260811 04:36:32.873148 5333 tablet.cc:2369] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:36:32.879436 5171 tablet.cc:2369] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 04:36:32 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422991, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:32.899147 4999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.891531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.2.238.190:46799)
0811 04:36:32.891670 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.891673 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.891865 (+ 192us) server_negotiation.cc:407] Connection header received
0811 04:36:32.891981 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.891983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.892024 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.892105 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:32.892730 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.893243 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.894028 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.894224 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.894307 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.896421 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:32.896424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:32.896426 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:32.896446 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:32.897600 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.898076 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.898079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.898080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.898151 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:32.898408 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:32.898410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:32.898411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:32.898572 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:32.898645 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.898914 (+ 269us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.899015 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":62}
I20260811 04:36:32.899155 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.891474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46799 (local address 127.0.0.1:40342)
0811 04:36:32.891769 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.891786 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.891903 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.892140 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.892142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.892514 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:32.892629 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.892635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.893330 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.893332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.893929 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.893938 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.894337 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.894339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.894398 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.894910 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:32.894921 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:32.896284 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:32.897694 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.897702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.897707 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.897993 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.898231 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:32.898233 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:32.898235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:32.898332 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:32.898660 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:32.898665 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:32.898765 (+ 100us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.898854 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.898962 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
I20260811 04:36:32.903327 4941 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:40342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 04:36:32.903999 4941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:32.908969 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 9030ea12b9f84531bf10c25075a2e03d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12d2d395e0d74a608c49bd6d9513803f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:32.908988 5098 tablet_service.cc:1467] Processing CreateTablet for tablet 9030ea12b9f84531bf10c25075a2e03d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12d2d395e0d74a608c49bd6d9513803f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:32.909312 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 9030ea12b9f84531bf10c25075a2e03d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=12d2d395e0d74a608c49bd6d9513803f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:36:32.909708 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030ea12b9f84531bf10c25075a2e03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.909725 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030ea12b9f84531bf10c25075a2e03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.909924 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9030ea12b9f84531bf10c25075a2e03d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:32.912221 5512 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: Bootstrap starting.
I20260811 04:36:32.912932 5511 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: Bootstrap starting.
I20260811 04:36:32.913260 5510 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: Bootstrap starting.
I20260811 04:36:32.913281 5512 tablet_bootstrap.cc:654] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.913952 5511 tablet_bootstrap.cc:654] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.914412 5510 tablet_bootstrap.cc:654] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:32.914724 5512 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: No bootstrap required, opened a new log
I20260811 04:36:32.914815 5512 ts_tablet_manager.cc:1397] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:32.915347 5511 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: No bootstrap required, opened a new log
I20260811 04:36:32.915298 5512 raft_consensus.cc:348] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.915412 5512 raft_consensus.cc:374] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.915416 5511 ts_tablet_manager.cc:1397] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 04:36:32.915462 5512 raft_consensus.cc:729] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.915601 5512 consensus_queue.cc:260] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.915894 5512 ts_tablet_manager.cc:1428] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:32.915863 5521 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.915891 5511 raft_consensus.cc:348] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.915992 5521 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.916000 5511 raft_consensus.cc:374] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.916031 5511 raft_consensus.cc:729] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.916155 5511 consensus_queue.cc:260] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.916471 5516 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.916622 5516 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.916775 5510 tablet_bootstrap.cc:492] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: No bootstrap required, opened a new log
I20260811 04:36:32.916872 5511 ts_tablet_manager.cc:1428] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:32.916887 5510 ts_tablet_manager.cc:1397] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 04:36:32.917382 5510 raft_consensus.cc:348] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.917539 5510 raft_consensus.cc:374] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:32.917574 5510 raft_consensus.cc:729] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Initialized, Role: FOLLOWER
I20260811 04:36:32.917702 5510 consensus_queue.cc:260] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.918008 5510 ts_tablet_manager.cc:1428] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:32.918017 5518 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.918200 5518 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.943780 5516 raft_consensus.cc:482] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:32.943935 5516 raft_consensus.cc:504] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.945047 5516 leader_election.cc:288] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071), 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279)
I20260811 04:36:32.949826 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.945321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:43071 (local address 127.2.238.131:43571)
0811 04:36:32.945524 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.945548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.945653 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.946010 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.946013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.946034 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.946194 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.946202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.947225 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.947227 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.948463 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.948474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.949348 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.949351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.949432 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.949480 (+ 48us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.949559 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.949636 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260811 04:36:32.949869 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.945460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:43571 (local address 127.2.238.129:43071)
0811 04:36:32.945609 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.945615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.945630 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:32.945844 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.945848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.945896 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.945993 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.946330 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.947113 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.948601 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.949227 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.949355 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.949416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.949607 (+ 191us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.949689 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:36:32.949970 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.945544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:55083 (local address 127.2.238.130:43279)
0811 04:36:32.945716 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.945721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.946038 (+ 317us) server_negotiation.cc:407] Connection header received
0811 04:36:32.946292 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.946295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.946348 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.946425 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:32.946774 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.947415 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.948777 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.949393 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.949508 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.949573 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.949760 (+ 187us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.949855 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260811 04:36:32.950026 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.945321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43279 (local address 127.2.238.131:55083)
0811 04:36:32.945945 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.945960 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.946115 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.946459 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.946462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.946480 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:32.946639 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.946651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.947524 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.947527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.948664 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.948671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.949519 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.949525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.949638 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.949658 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.949728 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.949812 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":141,"thread_start_us":156,"threads_started":1}
I20260811 04:36:32.950511 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030ea12b9f84531bf10c25075a2e03d" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716" is_pre_election: true
I20260811 04:36:32.950686 5120 raft_consensus.cc:2454] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 0.
I20260811 04:36:32.950641 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030ea12b9f84531bf10c25075a2e03d" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b" is_pre_election: true
I20260811 04:36:32.950922 5281 raft_consensus.cc:2454] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 0.
I20260811 04:36:32.951099 5363 leader_election.cc:302] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:32.951433 5516 raft_consensus.cc:2781] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:32.951494 5516 raft_consensus.cc:482] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:32.951522 5516 raft_consensus.cc:3037] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.952559 5516 raft_consensus.cc:504] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.952953 5516 leader_election.cc:288] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 election: Requested vote from peers 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071), 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279)
I20260811 04:36:32.953298 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030ea12b9f84531bf10c25075a2e03d" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b"
I20260811 04:36:32.953294 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9030ea12b9f84531bf10c25075a2e03d" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716"
I20260811 04:36:32.953401 5281 raft_consensus.cc:3037] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.953445 5120 raft_consensus.cc:3037] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:32.954406 5120 raft_consensus.cc:2454] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 1.
I20260811 04:36:32.954545 5281 raft_consensus.cc:2454] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 1.
I20260811 04:36:32.954785 5363 leader_election.cc:302] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:32.955056 5516 raft_consensus.cc:2781] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:32.955317 5516 raft_consensus.cc:686] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 LEADER]: Becoming Leader. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Running, Role: LEADER
I20260811 04:36:32.955742 5516 consensus_queue.cc:237] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:32.957948 5535 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: TxnStatusTablet state changed. Reason: New leader e218b0c523ad4881bbb95ab7b055a7d6. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.958060 5535 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:32.958366 5538 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 04:36:32.958581 5538 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 04:36:32.959223 4941 catalog_manager.cc:5495] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } health_report { overall_health: HEALTHY } } }
I20260811 04:36:32.976050 5530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.971798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.0.0.1:59292)
0811 04:36:32.972064 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:32.972077 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:32.972168 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:32.972500 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:32.972502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:32.972508 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:32.972601 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.972606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.973571 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.973573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.974222 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:32.974232 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.974780 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.974783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:32.974847 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.974893 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:32.975740 (+ 847us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:32.975754 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 04:36:32.975806 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 04:36:32.975901 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
I20260811 04:36:32.976071 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:32.971916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.2.238.131:41045)
0811 04:36:32.972060 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:32.972065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:32.972118 (+ 53us) server_negotiation.cc:407] Connection header received
0811 04:36:32.972308 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:32.972311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:32.972376 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:32.972487 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:32.972704 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.973462 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.974329 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:32.974661 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:32.974766 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:32.975031 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:32.975645 (+ 614us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:32.975754 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:32.975829 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 04:36:32.975907 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:36:32.981993 5281 raft_consensus.cc:1261] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Refusing update from remote peer e218b0c523ad4881bbb95ab7b055a7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:32.982040 5120 raft_consensus.cc:1261] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Refusing update from remote peer e218b0c523ad4881bbb95ab7b055a7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:32.982664 5535 consensus_queue.cc:1035] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:32.982856 5516 consensus_queue.cc:1035] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:32.988066 5518 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: TxnStatusTablet state changed. Reason: New leader e218b0c523ad4881bbb95ab7b055a7d6. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.988214 5518 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.989686 5521 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: TxnStatusTablet state changed. Reason: New leader e218b0c523ad4881bbb95ab7b055a7d6. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.989852 5521 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.990873 5516 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.990986 5516 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:32.991087 5516 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.991147 5516 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P e218b0c523ad4881bbb95ab7b055a7d6: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:36:32.991879 5542 mvcc.cc:205] Tried to move back new op lower bound from 7317188579246620672 to 7317188579149205504. Current Snapshot: MvccSnapshot[applied={T|T < 7317188579246620672}]
I20260811 04:36:32.992940 5521 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.993117 5521 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 14e0f678e82e4b30a8975ac0b228805b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.993247 5518 tablet_replica.cc:439] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } } }
I20260811 04:36:32.993366 5518 tablet_replica.cc:442] T 9030ea12b9f84531bf10c25075a2e03d P 77782ec70b62436587ca90ce3a578716: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:36:32.993611 5544 mvcc.cc:205] Tried to move back new op lower bound from 7317188579246620672 to 7317188579149205504. Current Snapshot: MvccSnapshot[applied={T|T < 7317188579246620672}]
I20260811 04:36:32.996179 5545 mvcc.cc:205] Tried to move back new op lower bound from 7317188579246620672 to 7317188579149205504. Current Snapshot: MvccSnapshot[applied={T|T < 7317188579246620672}]
I20260811 04:36:33.004745 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.000793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.2.238.130:43279)
0811 04:36:33.000931 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.000937 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.000953 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:36:33.001158 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.001161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.001226 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.001341 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.001705 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.002270 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.003212 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.003447 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.003577 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.003876 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.004284 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.004413 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.004514 (+ 101us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.004575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:36:33.016860 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.012962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:57273 (local address 127.2.238.131:41045)
0811 04:36:33.013096 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.013101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.013224 (+ 123us) server_negotiation.cc:407] Connection header received
0811 04:36:33.013341 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.013343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.013388 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.013455 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.013799 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.014421 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.015709 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.015919 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.016000 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.016424 (+ 424us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.016488 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.016565 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.016691 (+ 126us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.016742 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:36:33.016928 5556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.012794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.2.238.130:57273)
0811 04:36:33.013136 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.013151 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.013252 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.013486 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.013488 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.013496 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:33.013615 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.013620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.014533 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.014537 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.015500 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.015511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.016071 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.016074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.016168 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.016221 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.016598 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.016604 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.016669 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.016752 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":133,"threads_started":1}
I20260811 04:36:33.024394 5120 raft_consensus.cc:1261] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Refusing update from remote peer 14e0f678e82e4b30a8975ac0b228805b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:33.024530 5435 raft_consensus.cc:1261] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Refusing update from remote peer 14e0f678e82e4b30a8975ac0b228805b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:33.024889 5521 consensus_queue.cc:1035] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [LEADER]: Connected to new peer: Peer: permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:33.025067 5519 consensus_queue.cc:1035] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:33.049959 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.046245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:48669 (local address 127.2.238.130:43279)
0811 04:36:33.046349 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.046355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.046374 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:36:33.046503 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.046505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.046544 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.046605 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.046891 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.047381 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.048605 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.048934 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.049049 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.049452 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.049529 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.049585 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.049721 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.049772 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260811 04:36:33.049968 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.046101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43279 (local address 127.2.238.131:48669)
0811 04:36:33.046278 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.046294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.046382 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.046644 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.046646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.046654 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:33.046783 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.046790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.047493 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.047496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.048479 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.048490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.049149 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.049153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.049255 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.049322 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.049635 (+ 313us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.049642 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.049712 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.049817 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 04:36:33.075287 4944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:33.075500 4944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:33.078460 4944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Sending DeleteTablet for 3 replicas of tablet f51d698569f849a09887c670a9392120
I20260811 04:36:33.079188 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet f51d698569f849a09887c670a9392120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:33 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55998
I20260811 04:36:33.079430 5098 tablet_service.cc:1514] Processing DeleteTablet for tablet f51d698569f849a09887c670a9392120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:33 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:45230
I20260811 04:36:33.079876 5565 tablet_replica.cc:330] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: stopping tablet replica
I20260811 04:36:33.080252 5565 raft_consensus.cc:2229] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:33.080693 5565 raft_consensus.cc:2258] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:33.081080 4944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:33.081590 4944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:33.082531 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet f51d698569f849a09887c670a9392120 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:33 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:59278
I20260811 04:36:33.082849 5565 ts_tablet_manager.cc:1905] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:33.086196 5565 ts_tablet_manager.cc:1918] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:33.086378 5565 log.cc:1196] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/f51d698569f849a09887c670a9392120
I20260811 04:36:33.086755 5565 ts_tablet_manager.cc:1939] T f51d698569f849a09887c670a9392120 P 14e0f678e82e4b30a8975ac0b228805b: Deleting consensus metadata
I20260811 04:36:33.087740 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd00d5df13c4319b200982334f90be1 (DEFAULT_TABLE table=test-table [id=a8fca65da4804d5990fe80843aad4e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:33.088035 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd00d5df13c4319b200982334f90be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:33.088133 4929 catalog_manager.cc:4841] TS 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279): tablet f51d698569f849a09887c670a9392120 (table test-table [id=8e61e1d3079541c0872aa5733ed5a455]) successfully deleted
I20260811 04:36:33.090577 5512 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Bootstrap starting.
I20260811 04:36:33.090739 5567 tablet_replica.cc:330] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: stopping tablet replica
I20260811 04:36:33.091149 5567 raft_consensus.cc:2229] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:33.091409 5567 raft_consensus.cc:2258] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:33.091656 5512 tablet_bootstrap.cc:654] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:33.091867 5566 tablet_replica.cc:330] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: stopping tablet replica
I20260811 04:36:33.092244 5566 raft_consensus.cc:2229] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:33.092525 5566 raft_consensus.cc:2258] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:33.093011 5098 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd00d5df13c4319b200982334f90be1 (DEFAULT_TABLE table=test-table [id=a8fca65da4804d5990fe80843aad4e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:33.093360 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd00d5df13c4319b200982334f90be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:33.094178 5566 ts_tablet_manager.cc:1905] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:33.097137 5566 ts_tablet_manager.cc:1918] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:33.097297 5566 log.cc:1196] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/f51d698569f849a09887c670a9392120
I20260811 04:36:33.097656 5566 ts_tablet_manager.cc:1939] T f51d698569f849a09887c670a9392120 P 77782ec70b62436587ca90ce3a578716: Deleting consensus metadata
I20260811 04:36:33.098069 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 7cd00d5df13c4319b200982334f90be1 (DEFAULT_TABLE table=test-table [id=a8fca65da4804d5990fe80843aad4e54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:33.098423 5510 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Bootstrap starting.
I20260811 04:36:33.099092 4928 catalog_manager.cc:4841] TS 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071): tablet f51d698569f849a09887c670a9392120 (table test-table [id=8e61e1d3079541c0872aa5733ed5a455]) successfully deleted
I20260811 04:36:33.099459 5510 tablet_bootstrap.cc:654] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:33.100562 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cd00d5df13c4319b200982334f90be1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:33.103188 5511 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Bootstrap starting.
I20260811 04:36:33.104630 5511 tablet_bootstrap.cc:654] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:33.104627 5512 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: No bootstrap required, opened a new log
I20260811 04:36:33.104661 5567 ts_tablet_manager.cc:1905] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:33.104723 5512 ts_tablet_manager.cc:1397] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260811 04:36:33.105233 5512 raft_consensus.cc:348] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.105396 5512 raft_consensus.cc:374] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:33.105444 5512 raft_consensus.cc:729] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Initialized, Role: FOLLOWER
I20260811 04:36:33.105576 5512 consensus_queue.cc:260] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.105921 5512 ts_tablet_manager.cc:1428] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260811 04:36:33.107023 5510 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: No bootstrap required, opened a new log
I20260811 04:36:33.107132 5510 ts_tablet_manager.cc:1397] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260811 04:36:33.107597 5510 raft_consensus.cc:348] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.107713 5510 raft_consensus.cc:374] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:33.107686 5567 ts_tablet_manager.cc:1918] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:33.107745 5510 raft_consensus.cc:729] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Initialized, Role: FOLLOWER
I20260811 04:36:33.107792 5567 log.cc:1196] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/f51d698569f849a09887c670a9392120
I20260811 04:36:33.107867 5510 consensus_queue.cc:260] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.108174 5567 ts_tablet_manager.cc:1939] T f51d698569f849a09887c670a9392120 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting consensus metadata
I20260811 04:36:33.108206 5510 ts_tablet_manager.cc:1428] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:33.109496 4926 catalog_manager.cc:4841] TS e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045): tablet f51d698569f849a09887c670a9392120 (table test-table [id=8e61e1d3079541c0872aa5733ed5a455]) successfully deleted
I20260811 04:36:33.114712 5511 tablet_bootstrap.cc:492] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: No bootstrap required, opened a new log
I20260811 04:36:33.114814 5511 ts_tablet_manager.cc:1397] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260811 04:36:33.115296 5511 raft_consensus.cc:348] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.115422 5511 raft_consensus.cc:374] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:33.115466 5511 raft_consensus.cc:729] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Initialized, Role: FOLLOWER
I20260811 04:36:33.115607 5511 consensus_queue.cc:260] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.115983 5511 ts_tablet_manager.cc:1428] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:36:33.350984 5518 raft_consensus.cc:482] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:33.351157 5518 raft_consensus.cc:504] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.352419 5518 leader_election.cc:288] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279), e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045)
I20260811 04:36:33.357764 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.353055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:35677 (local address 127.2.238.131:41045)
0811 04:36:33.353181 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.353188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.353204 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:36:33.353415 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.353422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.353469 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.353550 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:33.353844 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.354617 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.356358 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.356997 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.357143 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.357209 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.357526 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.357614 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:33.357757 5506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.352728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:58765 (local address 127.2.238.130:43279)
0811 04:36:33.352892 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.352898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.353047 (+ 149us) server_negotiation.cc:407] Connection header received
0811 04:36:33.353304 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.353308 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.353374 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.353481 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:33.353762 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.354593 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.356358 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.356997 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.357145 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.357209 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.357526 (+ 317us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.357607 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 04:36:33.357919 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.352785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.2.238.129:35677)
0811 04:36:33.353133 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.353146 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.353265 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.353564 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.353567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.353584 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:33.353683 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.353689 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.354754 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.354759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.356223 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.356235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.357163 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.357167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.357360 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.357414 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.357488 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.357652 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":212,"thread_start_us":147,"threads_started":1}
I20260811 04:36:33.358112 5571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.352628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43279 (local address 127.2.238.129:58765)
0811 04:36:33.352942 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.352961 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.353155 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.353496 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.353499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.353517 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:33.353643 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.353649 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.354724 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.354730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.356223 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.356240 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.357163 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.357167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.357365 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.357410 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.357488 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.357652 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":216,"thread_start_us":92,"threads_started":1}
I20260811 04:36:33.358590 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd00d5df13c4319b200982334f90be1" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b" is_pre_election: true
I20260811 04:36:33.358618 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd00d5df13c4319b200982334f90be1" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" is_pre_election: true
I20260811 04:36:33.358726 5281 raft_consensus.cc:2454] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 0.
I20260811 04:36:33.358830 5435 raft_consensus.cc:2454] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 0.
I20260811 04:36:33.359206 5046 leader_election.cc:302] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 14e0f678e82e4b30a8975ac0b228805b, 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:33.359495 5518 raft_consensus.cc:2781] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:33.359557 5518 raft_consensus.cc:482] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:33.359596 5518 raft_consensus.cc:3037] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:33.360622 5518 raft_consensus.cc:504] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.360975 5518 leader_election.cc:288] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 election: Requested vote from peers 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279), e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045)
I20260811 04:36:33.361326 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd00d5df13c4319b200982334f90be1" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b"
I20260811 04:36:33.361409 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cd00d5df13c4319b200982334f90be1" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
I20260811 04:36:33.361464 5281 raft_consensus.cc:3037] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:33.361536 5435 raft_consensus.cc:3037] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:33.362787 5435 raft_consensus.cc:2454] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 1.
I20260811 04:36:33.362787 5281 raft_consensus.cc:2454] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 1.
I20260811 04:36:33.363123 5041 leader_election.cc:302] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:33.363282 5518 raft_consensus.cc:2781] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:33.363479 5518 raft_consensus.cc:686] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 1 LEADER]: Becoming Leader. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Running, Role: LEADER
I20260811 04:36:33.363741 5518 consensus_queue.cc:237] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:33.366039 4944 catalog_manager.cc:5495] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77782ec70b62436587ca90ce3a578716 (127.2.238.129). New cstate: current_term: 1 leader_uuid: "77782ec70b62436587ca90ce3a578716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:33.400981 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.397677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.2.238.129:43071)
0811 04:36:33.397798 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.397801 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.397833 (+ 32us) server_negotiation.cc:407] Connection header received
0811 04:36:33.397993 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.397996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.398055 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.398150 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.398333 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.398827 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.399684 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.399921 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.399995 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.400223 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.400673 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.400759 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.400828 (+ 69us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.400876 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:36:33.410434 5508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.407565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:37275 (local address 127.2.238.131:41045)
0811 04:36:33.407683 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.407686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.407807 (+ 121us) server_negotiation.cc:407] Connection header received
0811 04:36:33.407893 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.407895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.407932 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.407991 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.408206 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.408701 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.409500 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.409697 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.409769 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.410079 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.410140 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.410195 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.410279 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.410331 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 04:36:33.410463 5577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.407451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.2.238.129:37275)
0811 04:36:33.407736 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.407748 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.407829 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.408016 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.408018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.408023 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:33.408121 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.408125 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.408805 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.408808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.409404 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.409412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.409848 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.409851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.409923 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.409966 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.410217 (+ 251us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.410222 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.410272 (+ 50us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.410346 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":116,"threads_started":1}
I20260811 04:36:33.416147 5435 raft_consensus.cc:1261] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Refusing update from remote peer 77782ec70b62436587ca90ce3a578716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:33.416242 5281 raft_consensus.cc:1261] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Refusing update from remote peer 77782ec70b62436587ca90ce3a578716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:33.416587 5518 consensus_queue.cc:1035] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:33.416764 5573 consensus_queue.cc:1035] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:33.434034 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.430665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:43071 (local address 127.2.238.131:56299)
0811 04:36:33.430790 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:33.430805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:33.430888 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:33.431129 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:33.431132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:33.431139 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:33.431231 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.431237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.432031 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.432033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.432659 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:33.432668 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.433322 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.433325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:33.433387 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.433415 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.433734 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.433741 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 04:36:33.433825 (+ 84us) client_negotiation.cc:240] Negotiation successful
0811 04:36:33.433910 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260811 04:36:33.434494 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:33.430812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:56299 (local address 127.2.238.129:43071)
0811 04:36:33.430915 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:33.430922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:33.430937 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:33.430979 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:33.430982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:33.431022 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:33.431120 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:33.431388 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.431928 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.432789 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:33.433077 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:33.433161 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:33.433537 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:33.433628 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:33.433698 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:33.434294 (+ 596us) server_negotiation.cc:299] Negotiation successful
0811 04:36:33.434350 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 04:36:33.435217 5545 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 04:36:34.991214 4941 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:34.991355 4941 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:34.993444 4941 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Sending DeleteTablet for 3 replicas of tablet 7cd00d5df13c4319b200982334f90be1
I20260811 04:36:34.993932 5098 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd00d5df13c4319b200982334f90be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:34 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:45230
I20260811 04:36:34.993956 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd00d5df13c4319b200982334f90be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:34 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55998
I20260811 04:36:34.994038 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cd00d5df13c4319b200982334f90be1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:34 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:59278
I20260811 04:36:34.994254 5609 tablet_replica.cc:330] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: stopping tablet replica
I20260811 04:36:34.994334 5610 tablet_replica.cc:330] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: stopping tablet replica
I20260811 04:36:34.994375 5609 raft_consensus.cc:2229] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:34.994442 5610 raft_consensus.cc:2229] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:34.994455 5607 tablet_replica.cc:330] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: stopping tablet replica
I20260811 04:36:34.994472 5609 raft_consensus.cc:2258] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:34.994534 5610 raft_consensus.cc:2258] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:34.994550 5607 raft_consensus.cc:2229] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:34.994969 5607 raft_consensus.cc:2258] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:34.995054 5610 ts_tablet_manager.cc:1905] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:34.995078 5609 ts_tablet_manager.cc:1905] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:34.995512 5607 ts_tablet_manager.cc:1905] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:34.997537 5610 ts_tablet_manager.cc:1918] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 04:36:34.997605 5610 log.cc:1196] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/7cd00d5df13c4319b200982334f90be1
I20260811 04:36:34.997823 5609 ts_tablet_manager.cc:1918] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 04:36:34.997884 5609 log.cc:1196] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/7cd00d5df13c4319b200982334f90be1
I20260811 04:36:34.997898 5610 ts_tablet_manager.cc:1939] T 7cd00d5df13c4319b200982334f90be1 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting consensus metadata
I20260811 04:36:34.998046 5607 ts_tablet_manager.cc:1918] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 04:36:34.998104 5607 log.cc:1196] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/7cd00d5df13c4319b200982334f90be1
I20260811 04:36:34.998145 5609 ts_tablet_manager.cc:1939] T 7cd00d5df13c4319b200982334f90be1 P 14e0f678e82e4b30a8975ac0b228805b: Deleting consensus metadata
I20260811 04:36:34.998377 5607 ts_tablet_manager.cc:1939] T 7cd00d5df13c4319b200982334f90be1 P 77782ec70b62436587ca90ce3a578716: Deleting consensus metadata
I20260811 04:36:34.998587 4926 catalog_manager.cc:4841] TS e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045): tablet 7cd00d5df13c4319b200982334f90be1 (table test-table [id=a8fca65da4804d5990fe80843aad4e54]) successfully deleted
I20260811 04:36:34.998835 4929 catalog_manager.cc:4841] TS 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279): tablet 7cd00d5df13c4319b200982334f90be1 (table test-table [id=a8fca65da4804d5990fe80843aad4e54]) successfully deleted
I20260811 04:36:35.000396 4928 catalog_manager.cc:4841] TS 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071): tablet 7cd00d5df13c4319b200982334f90be1 (table test-table [id=a8fca65da4804d5990fe80843aad4e54]) successfully deleted
Aug 11 04:36:35 dist-test-slave-0kls krb5kdc[4892](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 04:36:35 dist-test-slave-0kls krb5kdc[4892](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422995, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:35.028579 5618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:35.016498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.2.238.190:46799)
0811 04:36:35.016762 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:35.016771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:35.016862 (+ 91us) server_negotiation.cc:407] Connection header received
0811 04:36:35.017051 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:35.017054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:35.017119 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:35.017214 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:35.017895 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.018492 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.019325 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.019561 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.019684 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:35.025520 (+ 5836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:35.025523 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:35.025525 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:35.025548 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:35.026895 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:35.027297 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:35.027305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:35.027306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:35.027382 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:35.027642 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:35.027645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:35.027647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:35.027831 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:35.027996 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:35.028217 (+ 221us) server_negotiation.cc:299] Negotiation successful
0811 04:36:35.028346 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260811 04:36:35.031076 4941 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:35.031522 4941 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:35.035902 5098 tablet_service.cc:1467] Processing CreateTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 (DEFAULT_TABLE table=test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.035982 5256 tablet_service.cc:1467] Processing CreateTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 (DEFAULT_TABLE table=test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.036252 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27d425a9084a4fbb5bd56fed3c0cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.036280 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27d425a9084a4fbb5bd56fed3c0cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.036257 5413 tablet_service.cc:1467] Processing CreateTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 (DEFAULT_TABLE table=test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.036513 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df27d425a9084a4fbb5bd56fed3c0cf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.038430 5619 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Bootstrap starting.
I20260811 04:36:35.039326 5619 tablet_bootstrap.cc:654] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.039429 5621 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Bootstrap starting.
I20260811 04:36:35.039440 5620 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Bootstrap starting.
I20260811 04:36:35.040298 5620 tablet_bootstrap.cc:654] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.040298 5621 tablet_bootstrap.cc:654] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.040524 5619 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: No bootstrap required, opened a new log
I20260811 04:36:35.040621 5619 ts_tablet_manager.cc:1397] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:35.041078 5619 raft_consensus.cc:348] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.041213 5619 raft_consensus.cc:374] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.041258 5619 raft_consensus.cc:729] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.041428 5619 consensus_queue.cc:260] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.041950 5619 ts_tablet_manager.cc:1428] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260811 04:36:35.041966 5620 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: No bootstrap required, opened a new log
I20260811 04:36:35.042042 5620 ts_tablet_manager.cc:1397] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:35.042302 5621 tablet_bootstrap.cc:492] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: No bootstrap required, opened a new log
I20260811 04:36:35.042409 5621 ts_tablet_manager.cc:1397] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:35.042464 5620 raft_consensus.cc:348] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.042578 5620 raft_consensus.cc:374] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.042614 5620 raft_consensus.cc:729] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.042737 5620 consensus_queue.cc:260] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.042822 5621 raft_consensus.cc:348] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.042946 5621 raft_consensus.cc:374] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.042994 5621 raft_consensus.cc:729] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.043017 5620 ts_tablet_manager.cc:1428] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 04:36:35.043148 5621 consensus_queue.cc:260] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.043797 5621 ts_tablet_manager.cc:1428] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:35.150741 5560 raft_consensus.cc:482] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:35.150887 5560 raft_consensus.cc:504] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.151269 5560 leader_election.cc:288] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071), 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279)
I20260811 04:36:35.151757 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27d425a9084a4fbb5bd56fed3c0cf7" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b" is_pre_election: true
I20260811 04:36:35.151757 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27d425a9084a4fbb5bd56fed3c0cf7" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716" is_pre_election: true
I20260811 04:36:35.151919 5281 raft_consensus.cc:2454] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 0.
I20260811 04:36:35.151919 5120 raft_consensus.cc:2454] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 0.
I20260811 04:36:35.152290 5363 leader_election.cc:302] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:35.152480 5560 raft_consensus.cc:2781] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:35.152536 5560 raft_consensus.cc:482] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:35.152590 5560 raft_consensus.cc:3037] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.153642 5560 raft_consensus.cc:504] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.153996 5560 leader_election.cc:288] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 election: Requested vote from peers 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071), 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279)
I20260811 04:36:35.154326 5120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27d425a9084a4fbb5bd56fed3c0cf7" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77782ec70b62436587ca90ce3a578716"
I20260811 04:36:35.154326 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df27d425a9084a4fbb5bd56fed3c0cf7" candidate_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b"
I20260811 04:36:35.154465 5281 raft_consensus.cc:3037] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.154465 5120 raft_consensus.cc:3037] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.155628 5281 raft_consensus.cc:2454] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 1.
I20260811 04:36:35.155635 5120 raft_consensus.cc:2454] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e218b0c523ad4881bbb95ab7b055a7d6 in term 1.
I20260811 04:36:35.155927 5364 leader_election.cc:302] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e0f678e82e4b30a8975ac0b228805b, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:35.156127 5560 raft_consensus.cc:2781] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:35.156215 5560 raft_consensus.cc:686] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 LEADER]: Becoming Leader. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Running, Role: LEADER
I20260811 04:36:35.156363 5560 consensus_queue.cc:237] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.158147 4942 catalog_manager.cc:5495] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } health_report { overall_health: HEALTHY } } }
I20260811 04:36:35.201138 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:35.197911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59304 (local address 127.2.238.131:41045)
0811 04:36:35.198179 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:35.198189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:35.198204 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:35.198244 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:35.198246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:35.198308 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:35.198396 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:35.198678 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.199201 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.200045 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.200279 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.200386 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:35.200637 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:35.200713 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:35.200801 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:35.200916 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 04:36:35.200969 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
I20260811 04:36:35.208920 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:35.206310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:52379 (local address 127.2.238.131:41045)
0811 04:36:35.206414 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:35.206418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:35.206442 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:36:35.206544 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:35.206546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:35.206581 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:35.206636 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:35.206868 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.207327 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.208071 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.208265 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.208335 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:35.208562 (+ 227us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:35.208619 (+ 57us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:35.208677 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:35.208752 (+ 75us) server_negotiation.cc:299] Negotiation successful
0811 04:36:35.208802 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":48}
I20260811 04:36:35.208936 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:35.206155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41045 (local address 127.2.238.131:52379)
0811 04:36:35.206386 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:35.206397 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:35.206473 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:35.206653 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:35.206654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:35.206660 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:36:35.206767 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:35.206771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.207409 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.207411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:35.207960 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:35.207985 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.208350 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.208352 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:35.208421 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:35.208462 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:36:35.208691 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:36:35.208696 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:36:35.208739 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 04:36:35.208805 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":13,"thread_start_us":84,"threads_started":1}
I20260811 04:36:35.213966 5281 raft_consensus.cc:1261] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Refusing update from remote peer e218b0c523ad4881bbb95ab7b055a7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:35.213980 5120 raft_consensus.cc:1261] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Refusing update from remote peer e218b0c523ad4881bbb95ab7b055a7d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:35.214355 5587 consensus_queue.cc:1035] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:35.214531 5592 consensus_queue.cc:1035] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:35.246272 4944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:35.246454 4944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:35.248351 4944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Sending DeleteTablet for 3 replicas of tablet df27d425a9084a4fbb5bd56fed3c0cf7
I20260811 04:36:35.248903 5098 tablet_service.cc:1514] Processing DeleteTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:35 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:45230
I20260811 04:36:35.248961 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:35 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55998
I20260811 04:36:35.249089 5607 tablet_replica.cc:330] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: stopping tablet replica
I20260811 04:36:35.249145 5609 tablet_replica.cc:330] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: stopping tablet replica
I20260811 04:36:35.249197 5607 raft_consensus.cc:2229] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:35.249264 5609 raft_consensus.cc:2229] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:35.249250 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet df27d425a9084a4fbb5bd56fed3c0cf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:35 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:59278
I20260811 04:36:35.249296 5607 raft_consensus.cc:2258] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:35.249372 5609 raft_consensus.cc:2258] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:35.249446 5610 tablet_replica.cc:330] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: stopping tablet replica
I20260811 04:36:35.249557 5610 raft_consensus.cc:2229] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:35.249943 5610 raft_consensus.cc:2258] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:35.250137 5609 ts_tablet_manager.cc:1905] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:35.250527 5607 ts_tablet_manager.cc:1905] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:35.250658 4944 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:35.251161 4944 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:35.252854 5609 ts_tablet_manager.cc:1918] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:35.252931 5609 log.cc:1196] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/df27d425a9084a4fbb5bd56fed3c0cf7
I20260811 04:36:35.253033 5607 ts_tablet_manager.cc:1918] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:35.253101 5607 log.cc:1196] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/df27d425a9084a4fbb5bd56fed3c0cf7
I20260811 04:36:35.253217 5609 ts_tablet_manager.cc:1939] T df27d425a9084a4fbb5bd56fed3c0cf7 P 14e0f678e82e4b30a8975ac0b228805b: Deleting consensus metadata
I20260811 04:36:35.253357 5607 ts_tablet_manager.cc:1939] T df27d425a9084a4fbb5bd56fed3c0cf7 P 77782ec70b62436587ca90ce3a578716: Deleting consensus metadata
I20260811 04:36:35.253985 4929 catalog_manager.cc:4841] TS 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279): tablet df27d425a9084a4fbb5bd56fed3c0cf7 (table test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]) successfully deleted
I20260811 04:36:35.254112 4928 catalog_manager.cc:4841] TS 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071): tablet df27d425a9084a4fbb5bd56fed3c0cf7 (table test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]) successfully deleted
I20260811 04:36:35.254580 5610 ts_tablet_manager.cc:1905] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:35.256422 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 (DEFAULT_TABLE table=test-table [id=f52632a0f5eb4a14b08ea1d91fb3ee28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.256439 5256 tablet_service.cc:1467] Processing CreateTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 (DEFAULT_TABLE table=test-table [id=f52632a0f5eb4a14b08ea1d91fb3ee28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.256677 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d07a9b5156c4a53a8e8c0c565713b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.256687 5256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d07a9b5156c4a53a8e8c0c565713b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.257053 5610 ts_tablet_manager.cc:1918] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:36:35.257112 5610 log.cc:1196] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/df27d425a9084a4fbb5bd56fed3c0cf7
I20260811 04:36:35.257333 5610 ts_tablet_manager.cc:1939] T df27d425a9084a4fbb5bd56fed3c0cf7 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting consensus metadata
I20260811 04:36:35.258301 4926 catalog_manager.cc:4841] TS e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045): tablet df27d425a9084a4fbb5bd56fed3c0cf7 (table test-table [id=6f4621d82c044291b6e15d7bc3ee4f2a]) successfully deleted
I20260811 04:36:35.258843 5621 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Bootstrap starting.
I20260811 04:36:35.258991 5619 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Bootstrap starting.
I20260811 04:36:35.259660 5621 tablet_bootstrap.cc:654] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.259683 5619 tablet_bootstrap.cc:654] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.260736 5621 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: No bootstrap required, opened a new log
I20260811 04:36:35.260823 5621 ts_tablet_manager.cc:1397] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:35.261178 5621 raft_consensus.cc:348] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.261283 5621 raft_consensus.cc:374] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.261313 5621 raft_consensus.cc:729] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e218b0c523ad4881bbb95ab7b055a7d6, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.261417 5621 consensus_queue.cc:260] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.261441 5098 tablet_service.cc:1467] Processing CreateTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 (DEFAULT_TABLE table=test-table [id=f52632a0f5eb4a14b08ea1d91fb3ee28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:35.261538 5619 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: No bootstrap required, opened a new log
I20260811 04:36:35.261610 5619 ts_tablet_manager.cc:1397] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:35.261654 5621 ts_tablet_manager.cc:1428] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:35.261680 5098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d07a9b5156c4a53a8e8c0c565713b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:35.262012 5619 raft_consensus.cc:348] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.262125 5619 raft_consensus.cc:374] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.262163 5619 raft_consensus.cc:729] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14e0f678e82e4b30a8975ac0b228805b, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.262264 5619 consensus_queue.cc:260] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.262471 5619 ts_tablet_manager.cc:1428] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:35.263813 5620 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: Bootstrap starting.
I20260811 04:36:35.264667 5620 tablet_bootstrap.cc:654] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:35.266537 5620 tablet_bootstrap.cc:492] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: No bootstrap required, opened a new log
I20260811 04:36:35.266626 5620 ts_tablet_manager.cc:1397] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:35.267028 5620 raft_consensus.cc:348] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.267135 5620 raft_consensus.cc:374] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:35.267177 5620 raft_consensus.cc:729] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Initialized, Role: FOLLOWER
I20260811 04:36:35.267304 5620 consensus_queue.cc:260] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.267534 5620 ts_tablet_manager.cc:1428] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:35.500717 5600 raft_consensus.cc:482] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:35.500873 5600 raft_consensus.cc:504] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.501324 5600 leader_election.cc:288] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279), e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045)
I20260811 04:36:35.501824 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d07a9b5156c4a53a8e8c0c565713b90" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" is_pre_election: true
I20260811 04:36:35.501843 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d07a9b5156c4a53a8e8c0c565713b90" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b" is_pre_election: true
I20260811 04:36:35.502009 5435 raft_consensus.cc:2454] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 0.
I20260811 04:36:35.502019 5281 raft_consensus.cc:2454] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 0.
I20260811 04:36:35.502341 5046 leader_election.cc:302] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14e0f678e82e4b30a8975ac0b228805b, 77782ec70b62436587ca90ce3a578716; no voters:
I20260811 04:36:35.502542 5600 raft_consensus.cc:2781] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:35.502601 5600 raft_consensus.cc:482] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:35.502629 5600 raft_consensus.cc:3037] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.503698 5600 raft_consensus.cc:504] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.504091 5600 leader_election.cc:288] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 election: Requested vote from peers 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279), e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045)
I20260811 04:36:35.504480 5281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d07a9b5156c4a53a8e8c0c565713b90" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14e0f678e82e4b30a8975ac0b228805b"
I20260811 04:36:35.504479 5435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d07a9b5156c4a53a8e8c0c565713b90" candidate_uuid: "77782ec70b62436587ca90ce3a578716" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e218b0c523ad4881bbb95ab7b055a7d6"
I20260811 04:36:35.504594 5281 raft_consensus.cc:3037] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.504663 5435 raft_consensus.cc:3037] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:35.505573 5281 raft_consensus.cc:2454] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 1.
I20260811 04:36:35.505636 5435 raft_consensus.cc:2454] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77782ec70b62436587ca90ce3a578716 in term 1.
I20260811 04:36:35.506141 5041 leader_election.cc:302] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77782ec70b62436587ca90ce3a578716, e218b0c523ad4881bbb95ab7b055a7d6; no voters:
I20260811 04:36:35.506425 5600 raft_consensus.cc:2781] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:35.506666 5600 raft_consensus.cc:686] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [term 1 LEADER]: Becoming Leader. State: Replica: 77782ec70b62436587ca90ce3a578716, State: Running, Role: LEADER
I20260811 04:36:35.506850 5600 consensus_queue.cc:237] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } }
I20260811 04:36:35.508589 4942 catalog_manager.cc:5495] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77782ec70b62436587ca90ce3a578716 (127.2.238.129). New cstate: current_term: 1 leader_uuid: "77782ec70b62436587ca90ce3a578716" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77782ec70b62436587ca90ce3a578716" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 43071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:35.564971 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:35.562013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45256 (local address 127.2.238.129:43071)
0811 04:36:35.562315 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:35.562322 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:35.562335 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:35.562374 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:35.562376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:35.562415 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:35.562486 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:35.562678 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.563253 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.563983 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:35.564218 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:35.564295 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:35.564497 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:35.564566 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:35.564618 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:35.564737 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 04:36:35.564782 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":139,"threads_started":1}
I20260811 04:36:35.571281 5435 raft_consensus.cc:1261] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Refusing update from remote peer 77782ec70b62436587ca90ce3a578716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:35.571382 5281 raft_consensus.cc:1261] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Refusing update from remote peer 77782ec70b62436587ca90ce3a578716: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:35.571715 5600 consensus_queue.cc:1035] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e218b0c523ad4881bbb95ab7b055a7d6" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:35.571892 5638 consensus_queue.cc:1035] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14e0f678e82e4b30a8975ac0b228805b" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 04:36:35.584908 5590 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260811 04:36:37.150161 4944 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:37.150318 4944 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:37.152789 4944 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26bb60cbaf5e40faa095c237192f2c29: Sending DeleteTablet for 3 replicas of tablet 4d07a9b5156c4a53a8e8c0c565713b90
I20260811 04:36:37.153349 5098 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:37 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:45230
I20260811 04:36:37.153434 5256 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:37 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55998
I20260811 04:36:37.153730 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d07a9b5156c4a53a8e8c0c565713b90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:37 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:59278
I20260811 04:36:37.153813 5670 tablet_replica.cc:330] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: stopping tablet replica
I20260811 04:36:37.153985 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5022
I20260811 04:36:37.154008 5669 tablet_replica.cc:330] T 4d07a9b5156c4a53a8e8c0c565713b90 P 77782ec70b62436587ca90ce3a578716: stopping tablet replica
I20260811 04:36:37.154019 5670 raft_consensus.cc:2229] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:37.154088 5671 tablet_replica.cc:330] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: stopping tablet replica
I20260811 04:36:37.156697 5671 raft_consensus.cc:2229] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:37.156842 5671 raft_consensus.cc:2258] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:37.157177 5670 raft_consensus.cc:2258] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:37.157593 5671 ts_tablet_manager.cc:1905] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:37.157915 5670 ts_tablet_manager.cc:1905] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:37.160107 5670 ts_tablet_manager.cc:1918] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 04:36:37.160202 5670 log.cc:1196] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/4d07a9b5156c4a53a8e8c0c565713b90
I20260811 04:36:37.160480 5670 ts_tablet_manager.cc:1939] T 4d07a9b5156c4a53a8e8c0c565713b90 P 14e0f678e82e4b30a8975ac0b228805b: Deleting consensus metadata
I20260811 04:36:37.161190 4929 catalog_manager.cc:4841] TS 14e0f678e82e4b30a8975ac0b228805b (127.2.238.130:43279): tablet 4d07a9b5156c4a53a8e8c0c565713b90 (table test-table [id=f52632a0f5eb4a14b08ea1d91fb3ee28]) successfully deleted
I20260811 04:36:37.161237 5671 ts_tablet_manager.cc:1918] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260811 04:36:37.161312 5671 log.cc:1196] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/4d07a9b5156c4a53a8e8c0c565713b90
I20260811 04:36:37.161638 5671 ts_tablet_manager.cc:1939] T 4d07a9b5156c4a53a8e8c0c565713b90 P e218b0c523ad4881bbb95ab7b055a7d6: Deleting consensus metadata
I20260811 04:36:37.162420 4926 catalog_manager.cc:4841] TS e218b0c523ad4881bbb95ab7b055a7d6 (127.2.238.131:41045): tablet 4d07a9b5156c4a53a8e8c0c565713b90 (table test-table [id=f52632a0f5eb4a14b08ea1d91fb3ee28]) successfully deleted
W20260811 04:36:37.166541 4928 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:43071 (error 108)
I20260811 04:36:37.166811 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5180
I20260811 04:36:37.169585 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:37.167820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:43071 (local address 127.0.0.1:45270)
0811 04:36:37.169356 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:37.169458 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:43071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1436,"thread_start_us":149,"threads_started":1}
W20260811 04:36:37.169936 4928 catalog_manager.cc:4584] TS 77782ec70b62436587ca90ce3a578716 (127.2.238.129:43071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d07a9b5156c4a53a8e8c0c565713b90: Network error: Client connection negotiation failed: client connection to 127.2.238.129:43071: connect: Connection refused (error 111)
I20260811 04:36:37.177675 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5342
I20260811 04:36:37.187882 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 4906
2026-08-11T04:36:37Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6774 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:37 dist-test-slave-0kls krb5kdc[5676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:37 dist-test-slave-0kls krb5kdc[5676](info): set up 2 sockets
Aug 11 04:36:37 dist-test-slave-0kls krb5kdc[5676](info): commencing operation
krb5kdc: starting...
W20260811 04:36:38.392414 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.178s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260811 04:36:38.536630 3002 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.144s user 0.000s sys 0.003s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260811 04:36:38.676913 3002 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.140s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260811 04:36:38.827535 3002 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.150s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/test-user.keytab.
W20260811 04:36:39.348549 3002 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.521s user 0.000s sys 0.002s
Aug 11 04:36:39 dist-test-slave-0kls krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
W20260811 04:36:39.640369 3002 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.238.190: real 0.264s user 0.000s sys 0.004s
I20260811 04:36:39.640928 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39447
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:39.758810 5692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:39.759093 5692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:39.759136 5692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:39.763190 5692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:39.763257 5692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:39.763276 5692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:39.763294 5692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:39.763309 5692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:39.768712 5692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:39.769901 5692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:39.770972 5692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:36:39.780355 5692 server_base.cc:1034] running on GCE node
W20260811 04:36:39.780319 5698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:39.780311 5699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:39.780311 5701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:39.781242 5692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:39.782541 5692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:39.783748 5692 hybrid_clock.cc:648] HybridClock initialized: now 1786422999783708 us; error 49 us; skew 500 ppm
Aug 11 04:36:39 dist-test-slave-0kls krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422999, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:39.786731 5692 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:39.787663 5692 webserver.cc:460] Webserver started at http://127.2.238.190:44591/ using document root <none> and password file <none>
I20260811 04:36:39.788293 5692 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:39.788378 5692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:39.788611 5692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:39.791006 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4"
format_stamp: "Formatted at 2026-08-11 04:36:39 on dist-test-slave-0kls"
server_key: "5a2725a9fde93439f2829260bb8fe579"
server_key_iv: "2c1d3dd81b612b6f944c4d63f776bbc6"
server_key_version: "encryptionkey@0"
I20260811 04:36:39.791656 5692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4"
format_stamp: "Formatted at 2026-08-11 04:36:39 on dist-test-slave-0kls"
server_key: "5a2725a9fde93439f2829260bb8fe579"
server_key_iv: "2c1d3dd81b612b6f944c4d63f776bbc6"
server_key_version: "encryptionkey@0"
I20260811 04:36:39.796063 5692 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:39.798609 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:39.799896 5692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:39.800042 5692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4"
format_stamp: "Formatted at 2026-08-11 04:36:39 on dist-test-slave-0kls"
server_key: "5a2725a9fde93439f2829260bb8fe579"
server_key_iv: "2c1d3dd81b612b6f944c4d63f776bbc6"
server_key_version: "encryptionkey@0"
I20260811 04:36:39.800163 5692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:39.817200 5692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:39.820343 5692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:39.820557 5692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:39.828045 5692 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39447
I20260811 04:36:39.828073 5765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39447 every 8 connection(s)
I20260811 04:36:39.829041 5692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:39.832070 5766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:39.837977 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4: Bootstrap starting.
I20260811 04:36:39.838428 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5692
I20260811 04:36:39.838546 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:39.838855 3002 external_mini_cluster.cc:1428] Setting key 700d0f83d7c31e13d8a8b84a91a5cf53
I20260811 04:36:39.841094 5766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:39.841930 5766 log.cc:824] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:39.844092 5766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4: No bootstrap required, opened a new log
Aug 11 04:36:39 dist-test-slave-0kls krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786422999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:39.848853 5766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } }
I20260811 04:36:39.849107 5766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:39.849185 5766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88f8de7fcd514cc2bfd2e23f8f8287e4, State: Initialized, Role: FOLLOWER
I20260811 04:36:39.849682 5766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } }
I20260811 04:36:39.849861 5766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:39.849960 5766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:39.850063 5766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:39.850992 5766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } }
I20260811 04:36:39.851377 5766 leader_election.cc:302] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88f8de7fcd514cc2bfd2e23f8f8287e4; no voters:
I20260811 04:36:39.851768 5766 leader_election.cc:288] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:39.851862 5771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:39.852089 5771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [term 1 LEADER]: Becoming Leader. State: Replica: 88f8de7fcd514cc2bfd2e23f8f8287e4, State: Running, Role: LEADER
I20260811 04:36:39.852501 5771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } }
I20260811 04:36:39.853204 5766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:39.854274 5774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88f8de7fcd514cc2bfd2e23f8f8287e4. Latest consensus state: current_term: 1 leader_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } } }
I20260811 04:36:39.854409 5774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:39.854645 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:39.840415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.2.238.190:39447)
0811 04:36:39.840894 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:39.840910 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:39.840946 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:36:39.841643 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:39.841662 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:39.842092 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:39.842457 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:39.843300 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:39.844326 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:39.845471 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:39.846007 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:39.846127 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:39.849104 (+ 2977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:39.849110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:39.849144 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:39.849175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:39.851091 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:39.851631 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:39.851638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:39.851644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:39.851791 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:39.852174 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:39.852177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:39.852179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:39.852679 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:39.852878 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:39.853197 (+ 319us) server_negotiation.cc:299] Negotiation successful
0811 04:36:39.853510 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":156,"threads_started":1}
I20260811 04:36:39.854812 5781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:39.854768 5772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88f8de7fcd514cc2bfd2e23f8f8287e4" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39447 } } }
I20260811 04:36:39.854854 5772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:39.858145 5781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:39.863426 5781 catalog_manager.cc:1347] Generated new cluster ID: 0f2aec353d94413aa4cae3dcfdfd1943
I20260811 04:36:39.863538 5781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:39.876248 5781 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:39.877549 5781 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:39.886826 5781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88f8de7fcd514cc2bfd2e23f8f8287e4: Generated new TSK 0
I20260811 04:36:39.887630 5781 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:39.917253 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:40.039577 5793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:40.039927 5793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:40.040006 5793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:40.044032 5793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:40.044128 5793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:40.044226 5793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:40.049540 5793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:40.050773 5793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:40.051862 5793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:40.062306 5799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:40.062279 5800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:40.062369 5793 server_base.cc:1034] running on GCE node
W20260811 04:36:40.062541 5802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:40.062976 5793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:40.063699 5793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:40.064888 5793 hybrid_clock.cc:648] HybridClock initialized: now 1786423000064859 us; error 40 us; skew 500 ppm
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:40.071902 5793 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:40.072919 5793 webserver.cc:460] Webserver started at http://127.2.238.129:36085/ using document root <none> and password file <none>
I20260811 04:36:40.073501 5793 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:40.073586 5793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:40.073907 5793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:40.076102 5793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "4c6f865971b04ccd9561b6afad3bda32"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "c03f4252ce9f3532a0b50cf42947247e"
server_key_iv: "89347c32779c9fbd139bd3274c5d26b6"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.076710 5793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "4c6f865971b04ccd9561b6afad3bda32"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "c03f4252ce9f3532a0b50cf42947247e"
server_key_iv: "89347c32779c9fbd139bd3274c5d26b6"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.080799 5793 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 04:36:40.083472 5810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.084815 5793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:36:40.084952 5793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "4c6f865971b04ccd9561b6afad3bda32"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "c03f4252ce9f3532a0b50cf42947247e"
server_key_iv: "89347c32779c9fbd139bd3274c5d26b6"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.085068 5793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:40.099390 5793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:40.102996 5793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:40.103242 5793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:40.103922 5793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:40.104969 5793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:40.105020 5793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.105062 5793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:40.105082 5793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.115144 5793 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:41549
I20260811 04:36:40.115167 5937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:41549 every 8 connection(s)
I20260811 04:36:40.116114 5793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:40.125168 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5793
I20260811 04:36:40.125294 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:40.125648 3002 external_mini_cluster.cc:1428] Setting key ea156878e4b51f188a9f26de036d0e54
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:40.131083 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.118178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:46845 (local address 127.2.238.190:39447)
0811 04:36:40.118323 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:40.118327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:40.119225 (+ 898us) server_negotiation.cc:407] Connection header received
0811 04:36:40.120652 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:40.120656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:40.120718 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:40.120823 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:40.122063 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.122606 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.123738 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.123974 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.124093 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.127042 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:40.127044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:40.127046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:40.127097 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:40.128635 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.129323 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.129326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.129327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.129417 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.129856 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.129858 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.129859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.130063 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:40.130142 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:40.130816 (+ 674us) server_negotiation.cc:299] Negotiation successful
0811 04:36:40.130946 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:36:40.132359 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.118586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39447 (local address 127.2.238.129:46845)
0811 04:36:40.119065 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:40.119117 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:40.120423 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:40.121009 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:40.121016 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:40.121386 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:40.121913 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.121928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.122738 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.122741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.123614 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.123624 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.124132 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.124135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.124305 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.124887 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:40.124910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:40.126875 (+ 1965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:40.128815 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.128822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.128839 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.129210 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.129513 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.129519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.129522 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.129735 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.130166 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:40.130176 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:40.130535 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 04:36:40.130769 (+ 234us) client_negotiation.cc:240] Negotiation successful
0811 04:36:40.131179 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":132,"threads_started":1}
I20260811 04:36:40.133760 5938 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39447
I20260811 04:36:40.134083 5938 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:40.134758 5938 heartbeater.cc:511] Master 127.2.238.190:39447 requested a full tablet report, sending...
I20260811 04:36:40.136394 5727 ts_manager.cc:194] Registered new tserver with Master: 4c6f865971b04ccd9561b6afad3bda32 (127.2.238.129:41549)
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:40.137609 5727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:46845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:40.155027 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:40.270008 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:40.270354 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:40.270429 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:40.274471 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:40.274578 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:40.274674 5944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:40.280022 5944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:40.281195 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:40.282307 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:40.292115 5954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:40.292133 5952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:40.292119 5951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:40.292172 5944 server_base.cc:1034] running on GCE node
I20260811 04:36:40.292852 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:40.293540 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:40.294724 5944 hybrid_clock.cc:648] HybridClock initialized: now 1786423000294695 us; error 37 us; skew 500 ppm
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:40.297354 5944 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:40.298297 5944 webserver.cc:460] Webserver started at http://127.2.238.130:42451/ using document root <none> and password file <none>
I20260811 04:36:40.298851 5944 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:40.298910 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:40.299083 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:40.301136 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "bd36d1d1775e4c3096118de02a5bb79b"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "ba93a28e74453e3ae4571aa385c7015d"
server_key_iv: "beba32c7c3195f412a638099dc97fb27"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.301644 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "bd36d1d1775e4c3096118de02a5bb79b"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "ba93a28e74453e3ae4571aa385c7015d"
server_key_iv: "beba32c7c3195f412a638099dc97fb27"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.305635 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260811 04:36:40.308229 5961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.309518 5944 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:40.309624 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "bd36d1d1775e4c3096118de02a5bb79b"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "ba93a28e74453e3ae4571aa385c7015d"
server_key_iv: "beba32c7c3195f412a638099dc97fb27"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.309717 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:40.337512 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:40.340637 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:40.340816 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:40.341403 5944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:40.342291 5944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:40.342340 5944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.342434 5944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:40.342468 5944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.351476 5944 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43271
I20260811 04:36:40.351501 6088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43271 every 8 connection(s)
I20260811 04:36:40.352418 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:40.352774 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 5944
I20260811 04:36:40.352893 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:40.353250 3002 external_mini_cluster.cc:1428] Setting key 90b988a45e6f1410ce7d3089afed2b77
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
I20260811 04:36:40.369127 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.354457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:49377 (local address 127.2.238.190:39447)
0811 04:36:40.354587 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:40.354590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:40.355719 (+ 1129us) server_negotiation.cc:407] Connection header received
0811 04:36:40.357720 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:40.357726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:40.357829 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:40.357930 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:40.359409 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.359951 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.361917 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.362136 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.362219 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.365150 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:40.365153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:40.365155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:40.365201 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:40.366582 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.367341 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.367345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.367346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.367476 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.367850 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.367856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.367861 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.368087 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:40.368198 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:40.368864 (+ 666us) server_negotiation.cc:299] Negotiation successful
0811 04:36:40.368992 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:36:40.370505 6091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.354968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39447 (local address 127.2.238.130:49377)
0811 04:36:40.355531 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:40.355594 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:40.357493 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:40.358095 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:40.358102 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:40.358488 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:40.359226 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.359251 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.360089 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.360096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.361754 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.361770 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.362325 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.362332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.362582 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.363246 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:40.363279 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:40.364962 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:40.366713 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.366720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.366753 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.367164 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.367576 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.367579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.367581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.367751 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.368197 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:40.368206 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:40.368550 (+ 344us) client_negotiation.cc:769] Sending connection context
0811 04:36:40.368802 (+ 252us) client_negotiation.cc:240] Negotiation successful
0811 04:36:40.369223 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":113,"threads_started":1}
I20260811 04:36:40.371948 6089 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39447
I20260811 04:36:40.372210 6089 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:40.372761 6089 heartbeater.cc:511] Master 127.2.238.190:39447 requested a full tablet report, sending...
I20260811 04:36:40.374001 5727 ts_manager.cc:194] Registered new tserver with Master: bd36d1d1775e4c3096118de02a5bb79b (127.2.238.130:43271)
I20260811 04:36:40.374586 5727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:49377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:40.384868 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:45269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:40.510316 6094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:40.510627 6094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:40.510701 6094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:40.515017 6094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:40.515110 6094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:40.515201 6094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:40.520861 6094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:45269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39447
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:40.522205 6094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:40.523329 6094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:40.533756 6099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:40.533818 6103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:40.533818 6101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:40.534056 6094 server_base.cc:1034] running on GCE node
I20260811 04:36:40.534520 6094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:40.535197 6094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:40.536392 6094 hybrid_clock.cc:648] HybridClock initialized: now 1786423000536373 us; error 40 us; skew 500 ppm
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:40.539222 6094 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:40.540162 6094 webserver.cc:460] Webserver started at http://127.2.238.131:45791/ using document root <none> and password file <none>
I20260811 04:36:40.540763 6094 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:40.540856 6094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:40.541102 6094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:40.543304 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "1bfa23df102243878d8e63d34b2d93f2"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "f774d249594529d60be4aecb754b95d7"
server_key_iv: "a5ed62091c1d5a070079f84a78ba9e52"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.543906 6094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "1bfa23df102243878d8e63d34b2d93f2"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "f774d249594529d60be4aecb754b95d7"
server_key_iv: "a5ed62091c1d5a070079f84a78ba9e52"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.548135 6094 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:40.550742 6110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.552044 6094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:40.552217 6094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "1bfa23df102243878d8e63d34b2d93f2"
format_stamp: "Formatted at 2026-08-11 04:36:40 on dist-test-slave-0kls"
server_key: "f774d249594529d60be4aecb754b95d7"
server_key_iv: "a5ed62091c1d5a070079f84a78ba9e52"
server_key_version: "encryptionkey@0"
I20260811 04:36:40.552341 6094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:40.588533 6094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:40.592005 6094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:40.592262 6094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:40.592928 6094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:40.593906 6094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:40.593973 6094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.594048 6094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:40.594085 6094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:40.603310 6094 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:43785
I20260811 04:36:40.603332 6232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:43785 every 8 connection(s)
I20260811 04:36:40.604367 6094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:40.613528 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6094
I20260811 04:36:40.613672 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:40.614112 3002 external_mini_cluster.cc:1428] Setting key dd5ef863736f03fc21ce84e15f61bffd
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[5676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423000, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:40.618952 5769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.606385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40899 (local address 127.2.238.190:39447)
0811 04:36:40.606531 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:40.606535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:40.607421 (+ 886us) server_negotiation.cc:407] Connection header received
0811 04:36:40.608733 (+ 1312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:40.608736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:40.608798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:40.608875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:40.610166 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.610687 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.611828 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.612051 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.612129 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.614992 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:40.615015 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:40.615019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:40.615055 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:40.616683 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.617286 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.617290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.617292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.617386 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:40.617754 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:40.617758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:40.617759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:40.618003 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:40.618108 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:40.618644 (+ 536us) server_negotiation.cc:299] Negotiation successful
0811 04:36:40.618788 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:36:40.619817 6235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:40.606796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39447 (local address 127.2.238.131:40899)
0811 04:36:40.607258 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:40.607303 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:40.608530 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:40.609049 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:40.609056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:40.609431 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:40.610022 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.610041 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.610794 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.610797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.611701 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:40.611710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:40.612188 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:40.612191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:40.612351 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:40.612928 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:40.612982 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:40.614811 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:40.616815 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.616822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.616841 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.617145 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.617481 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:40.617484 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:40.617486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:40.617638 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:40.618123 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:40.618132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:40.618403 (+ 271us) client_negotiation.cc:769] Sending connection context
0811 04:36:40.618602 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 04:36:40.618953 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
I20260811 04:36:40.620940 6233 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39447
I20260811 04:36:40.621227 6233 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:40.621766 6233 heartbeater.cc:511] Master 127.2.238.190:39447 requested a full tablet report, sending...
I20260811 04:36:40.622890 5727 ts_manager.cc:194] Registered new tserver with Master: 1bfa23df102243878d8e63d34b2d93f2 (127.2.238.131:43785)
I20260811 04:36:40.623477 5727 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:40899
I20260811 04:36:40.628376 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 04:36:40.643321 5769 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:40.641586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.2.238.190:39447)
0811 04:36:40.641866 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:40.641872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:40.641994 (+ 122us) server_negotiation.cc:407] Connection header received
0811 04:36:40.642175 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:40.642178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:40.642231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:40.642318 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:40.643145 (+ 827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54532: BlockingRecv error: recv got EOF from 127.0.0.1:54532 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260811 04:36:40.644404 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5793
I20260811 04:36:40.655249 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5944
I20260811 04:36:40.665143 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6094
I20260811 04:36:40.673034 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 5692
2026-08-11T04:36:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3484 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[6245](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[6245](info): set up 2 sockets
Aug 11 04:36:40 dist-test-slave-0kls krb5kdc[6245](info): commencing operation
krb5kdc: starting...
W20260811 04:36:41.754961 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:41 dist-test-slave-0kls krb5kdc[6245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:41.882241 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34375
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:41.998531 6259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:41.998847 6259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:41.998961 6259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:42.004611 6259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:42.004711 6259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:42.004746 6259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:42.004777 6259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:42.004817 6259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:42.012560 6259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34375
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:42.013656 6259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:42.014781 6259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:42.023636 6265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:42.023815 6264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:42.023903 6259 server_base.cc:1034] running on GCE node
W20260811 04:36:42.023640 6267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:42.024894 6259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:42.026197 6259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:42.027387 6259 hybrid_clock.cc:648] HybridClock initialized: now 1786423002027374 us; error 26 us; skew 500 ppm
Aug 11 04:36:42 dist-test-slave-0kls krb5kdc[6245](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423002, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:42.030435 6259 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:42.031425 6259 webserver.cc:460] Webserver started at http://127.2.238.190:42973/ using document root <none> and password file <none>
I20260811 04:36:42.032001 6259 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:42.032111 6259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:42.032352 6259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:42.034616 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "634b34253077419abf22e07296c5255c"
format_stamp: "Formatted at 2026-08-11 04:36:42 on dist-test-slave-0kls"
server_key: "7b81b063d231b80f8e6d092123039f96"
server_key_iv: "6c4baea84f713997d9a1f50240566713"
server_key_version: "encryptionkey@0"
I20260811 04:36:42.035197 6259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "634b34253077419abf22e07296c5255c"
format_stamp: "Formatted at 2026-08-11 04:36:42 on dist-test-slave-0kls"
server_key: "7b81b063d231b80f8e6d092123039f96"
server_key_iv: "6c4baea84f713997d9a1f50240566713"
server_key_version: "encryptionkey@0"
I20260811 04:36:42.039318 6259 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:42.042277 6275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:42.043514 6259 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:42.043664 6259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "634b34253077419abf22e07296c5255c"
format_stamp: "Formatted at 2026-08-11 04:36:42 on dist-test-slave-0kls"
server_key: "7b81b063d231b80f8e6d092123039f96"
server_key_iv: "6c4baea84f713997d9a1f50240566713"
server_key_version: "encryptionkey@0"
I20260811 04:36:42.043792 6259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:42.055464 6259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:42.058774 6259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:42.058988 6259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:42.066479 6259 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34375
I20260811 04:36:42.066790 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34375 every 8 connection(s)
I20260811 04:36:42.067507 6259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:42.070035 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6259
I20260811 04:36:42.070163 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:42.070477 3002 external_mini_cluster.cc:1428] Setting key 51ab9a49f81b9225a447230b0929b5bc
I20260811 04:36:42.071091 6333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:42.078281 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: Bootstrap starting.
Aug 11 04:36:42 dist-test-slave-0kls krb5kdc[6245](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:42.081110 6333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:42.082093 6333 log.cc:824] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:42.084568 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: No bootstrap required, opened a new log
I20260811 04:36:42.085626 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:42.071869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.2.238.190:34375)
0811 04:36:42.072330 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:42.072347 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:42.072384 (+ 37us) server_negotiation.cc:407] Connection header received
0811 04:36:42.073047 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:42.073064 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:42.073407 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:42.073713 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:42.074753 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:42.075724 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:42.076956 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:42.077427 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:42.077539 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:42.080363 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:42.080370 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:42.080387 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:42.080420 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:42.082438 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:42.082942 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:42.082966 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:42.082975 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:42.083117 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:42.083381 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:42.083384 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:42.083385 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:42.083899 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:42.084087 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:42.084287 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 04:36:42.084610 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":146,"threads_started":1}
I20260811 04:36:42.088841 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:36:42.089236 6333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } }
I20260811 04:36:42.089500 6333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:42.089583 6333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 634b34253077419abf22e07296c5255c, State: Initialized, Role: FOLLOWER
I20260811 04:36:42.090080 6333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } }
I20260811 04:36:42.090258 6333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:42.090341 6333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:42.090444 6333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:42.091410 6333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } }
I20260811 04:36:42.091761 6333 leader_election.cc:302] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 634b34253077419abf22e07296c5255c; no voters:
I20260811 04:36:42.092095 6333 leader_election.cc:288] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:42.092201 6339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:42.092379 6339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [term 1 LEADER]: Becoming Leader. State: Replica: 634b34253077419abf22e07296c5255c, State: Running, Role: LEADER
I20260811 04:36:42.092685 6339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } }
I20260811 04:36:42.093464 6333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:42.094349 6340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "634b34253077419abf22e07296c5255c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } } }
I20260811 04:36:42.094506 6340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:42.094777 6340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 634b34253077419abf22e07296c5255c. Latest consensus state: current_term: 1 leader_uuid: "634b34253077419abf22e07296c5255c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "634b34253077419abf22e07296c5255c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34375 } } }
I20260811 04:36:42.094902 6340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c [sys.catalog]: This master's current role is: LEADER
W20260811 04:36:42.098963 6354 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:36:42.099088 6354 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:36:42.099223 6348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:42.099989 6348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:42.105907 6348 catalog_manager.cc:1347] Generated new cluster ID: 49947007c6e141118532493acb2362ef
I20260811 04:36:42.105976 6348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:42.114384 6336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:42.108494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.2.238.190:34375)
0811 04:36:42.108605 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:42.108610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:42.108901 (+ 291us) server_negotiation.cc:407] Connection header received
0811 04:36:42.109025 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:42.109030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:42.109091 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:42.109186 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:42.109824 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:42.110675 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:42.111823 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:42.112186 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:42.112314 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:42.113248 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:42.113252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:42.113256 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:42.113898 (+ 642us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:42.113938 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:42.113961 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:42.114053 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:42.114148 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 04:36:42.114219 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:36:42.127684 6348 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:42.128718 6348 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:42.136075 6348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 634b34253077419abf22e07296c5255c: Generated new TSK 0
I20260811 04:36:42.136826 6348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:36:42.144860 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6259
2026-08-11T04:36:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1469 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:42 dist-test-slave-0kls krb5kdc[6367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:42 dist-test-slave-0kls krb5kdc[6367](info): set up 2 sockets
Aug 11 04:36:42 dist-test-slave-0kls krb5kdc[6367](info): commencing operation
krb5kdc: starting...
W20260811 04:36:43.224897 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:43 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:43.356236 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42311
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:42311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:43.473701 6380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:43.474064 6380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:43.474174 6380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:43.478258 6380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:43.478370 6380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:43.478408 6380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:43.478448 6380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:43.478484 6380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:43.484107 6380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:42311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:43.485411 6380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:43.486553 6380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:43.495623 6385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:43.495785 6386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:43.495643 6388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:43.496013 6380 server_base.cc:1034] running on GCE node
I20260811 04:36:43.496651 6380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:43.497920 6380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:43.499121 6380 hybrid_clock.cc:648] HybridClock initialized: now 1786423003499085 us; error 47 us; skew 500 ppm
Aug 11 04:36:43 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:43.502200 6380 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:43.503201 6380 webserver.cc:460] Webserver started at http://127.2.238.190:39729/ using document root <none> and password file <none>
I20260811 04:36:43.503798 6380 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:43.503865 6380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:43.504061 6380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:43.506379 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "ab3efacbb2624e5fa83f7122c4139b2f"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "0adbb1988df8efe6389d55a993a90509"
server_key_iv: "7d161b55e214737a3ab356f13a6b9520"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.506956 6380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "ab3efacbb2624e5fa83f7122c4139b2f"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "0adbb1988df8efe6389d55a993a90509"
server_key_iv: "7d161b55e214737a3ab356f13a6b9520"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.511276 6380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:43.513999 6395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:43.515367 6380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:36:43.515540 6380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "ab3efacbb2624e5fa83f7122c4139b2f"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "0adbb1988df8efe6389d55a993a90509"
server_key_iv: "7d161b55e214737a3ab356f13a6b9520"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.515678 6380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:43.536932 6380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:43.540208 6380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:43.540447 6380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:43.548337 6380 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:42311
I20260811 04:36:43.548631 6449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:42311 every 8 connection(s)
I20260811 04:36:43.549450 6380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:43.552423 6450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:43.553987 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6380
I20260811 04:36:43.554116 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:43.554450 3002 external_mini_cluster.cc:1428] Setting key 20f19bb2a7d2c5cc12b77f83b9832f23
I20260811 04:36:43.559293 6450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f: Bootstrap starting.
I20260811 04:36:43.562184 6450 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:43.563225 6450 log.cc:824] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:36:43 dist-test-slave-0kls krb5kdc[6367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:43.565397 6450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f: No bootstrap required, opened a new log
I20260811 04:36:43.570206 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:43.555876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53020 (local address 127.2.238.190:42311)
0811 04:36:43.556320 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:43.556336 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:43.556374 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:36:43.557004 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:43.557021 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:43.557366 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:43.557676 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:43.558648 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.559682 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.560883 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.561373 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.561507 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:43.564550 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:43.564555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:43.564569 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:43.564597 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:43.566667 (+ 2070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:43.567135 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:43.567142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:43.567148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:43.567297 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:43.567649 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:43.567654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:43.567656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:43.568295 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:43.568487 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:43.568775 (+ 288us) server_negotiation.cc:299] Negotiation successful
0811 04:36:43.569112 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":160,"threads_started":1}
I20260811 04:36:43.570386 6450 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } }
I20260811 04:36:43.570673 6450 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:43.570747 6450 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab3efacbb2624e5fa83f7122c4139b2f, State: Initialized, Role: FOLLOWER
I20260811 04:36:43.571246 6450 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } }
I20260811 04:36:43.571416 6450 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:43.571484 6450 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:43.571607 6450 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:43.572670 6450 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } }
I20260811 04:36:43.573032 6450 leader_election.cc:302] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab3efacbb2624e5fa83f7122c4139b2f; no voters:
I20260811 04:36:43.573412 6450 leader_election.cc:288] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:43.573575 6456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:43.573848 6456 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [term 1 LEADER]: Becoming Leader. State: Replica: ab3efacbb2624e5fa83f7122c4139b2f, State: Running, Role: LEADER
I20260811 04:36:43.574173 6456 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } }
I20260811 04:36:43.574769 6450 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:43.576071 6458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab3efacbb2624e5fa83f7122c4139b2f. Latest consensus state: current_term: 1 leader_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } } }
I20260811 04:36:43.576088 6457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab3efacbb2624e5fa83f7122c4139b2f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42311 } } }
I20260811 04:36:43.576267 6458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:43.576292 6457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:43.576879 6466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:43.580334 6466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:43.585820 6466 catalog_manager.cc:1347] Generated new cluster ID: 0c588f82651642749f7c5ebe856ae309
I20260811 04:36:43.585892 6466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:43.622010 6466 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:43.623312 6466 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:43.628334 6466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab3efacbb2624e5fa83f7122c4139b2f: Generated new TSK 0
I20260811 04:36:43.629078 6466 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:43.672653 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:43.801462 6479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:43.801733 6479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:43.801777 6479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:43.805825 6479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:43.805903 6479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:43.805965 6479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:43.811375 6479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:43.812470 6479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:43.813464 6479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:43.823489 6487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:43.823501 6484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:43.823489 6485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:43.823951 6479 server_base.cc:1034] running on GCE node
I20260811 04:36:43.824362 6479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:43.825050 6479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:43.826234 6479 hybrid_clock.cc:648] HybridClock initialized: now 1786423003826217 us; error 30 us; skew 500 ppm
Aug 11 04:36:43 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:43.828986 6479 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:43.829946 6479 webserver.cc:460] Webserver started at http://127.2.238.129:39931/ using document root <none> and password file <none>
I20260811 04:36:43.830549 6479 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:43.830641 6479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:43.830883 6479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:43.833392 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "43893332c89144568ac52c3b0dac5191"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "b32ca5d80e64fb030ba0b2495dd6dcde"
server_key_iv: "b3ef45bd53fdfd85f32af6005ebc1c5e"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.834041 6479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "43893332c89144568ac52c3b0dac5191"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "b32ca5d80e64fb030ba0b2495dd6dcde"
server_key_iv: "b3ef45bd53fdfd85f32af6005ebc1c5e"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.838194 6479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:36:43.840873 6494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:43.842226 6479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:43.842366 6479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "43893332c89144568ac52c3b0dac5191"
format_stamp: "Formatted at 2026-08-11 04:36:43 on dist-test-slave-0kls"
server_key: "b32ca5d80e64fb030ba0b2495dd6dcde"
server_key_iv: "b3ef45bd53fdfd85f32af6005ebc1c5e"
server_key_version: "encryptionkey@0"
I20260811 04:36:43.842485 6479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:43.869103 6479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:43.872290 6479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:43.872509 6479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:43.873489 6479 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:43.874688 6479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:43.874758 6479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:43.874859 6479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:43.874907 6479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:43.884601 6479 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:45165
I20260811 04:36:43.884647 6611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:45165 every 8 connection(s)
I20260811 04:36:43.885574 6479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:43.891654 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6479
I20260811 04:36:43.891803 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:43.892118 3002 external_mini_cluster.cc:1428] Setting key 99068ff2244ed129218a986377fcf6f4
Aug 11 04:36:43 dist-test-slave-0kls krb5kdc[6367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423003, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:43.900774 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:43.887614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:45693 (local address 127.2.238.190:42311)
0811 04:36:43.887798 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:43.887802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:43.888625 (+ 823us) server_negotiation.cc:407] Connection header received
0811 04:36:43.890178 (+ 1553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:43.890182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:43.890246 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:43.890331 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:43.891579 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.892332 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.893659 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.893989 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.894092 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:43.896595 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:43.896598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:43.896600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:43.896623 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:43.898273 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:43.898877 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:43.898883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:43.898886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:43.899011 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:43.899371 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:43.899377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:43.899380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:43.899677 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:43.899808 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:43.900350 (+ 542us) server_negotiation.cc:299] Negotiation successful
0811 04:36:43.900514 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260811 04:36:43.901506 6614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:43.887990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42311 (local address 127.2.238.129:45693)
0811 04:36:43.888459 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:43.888505 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:43.889935 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:43.890515 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:43.890522 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:43.890903 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:43.891427 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:43.891441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.892475 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.892477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:43.893532 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:43.893542 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:43.894158 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:43.894162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:43.894339 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:43.894960 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:43.894990 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:43.896423 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:43.898409 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:43.898414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:43.898428 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:43.898743 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:43.899115 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:43.899117 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:43.899118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:43.899271 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:43.899813 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:43.899819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:43.900079 (+ 260us) client_negotiation.cc:769] Sending connection context
0811 04:36:43.900264 (+ 185us) client_negotiation.cc:240] Negotiation successful
0811 04:36:43.900536 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":130,"threads_started":1}
I20260811 04:36:43.902889 6612 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42311
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:43.903142 6612 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:43.903640 6612 heartbeater.cc:511] Master 127.2.238.190:42311 requested a full tablet report, sending...
I20260811 04:36:43.905419 6412 ts_manager.cc:194] Registered new tserver with Master: 43893332c89144568ac52c3b0dac5191 (127.2.238.129:45165)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:36:43.907138 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:45693
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:43.923101 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:44.043835 6618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:44.044112 6618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:44.044155 6618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:44.048094 6618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:44.048163 6618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:44.048226 6618 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:44.053515 6618 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6618
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:44.054716 6618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:44.055717 6618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:44.065590 6623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:44.065702 6627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:44.065590 6624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:44.066054 6618 server_base.cc:1034] running on GCE node
I20260811 04:36:44.066469 6618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:44.067091 6618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:44.068282 6618 hybrid_clock.cc:648] HybridClock initialized: now 1786423004068265 us; error 32 us; skew 500 ppm
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:44.071303 6618 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:44.072211 6618 webserver.cc:460] Webserver started at http://127.2.238.130:38467/ using document root <none> and password file <none>
I20260811 04:36:44.072810 6618 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:44.072916 6618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:44.073158 6618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:44.075381 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "a5bca2680d9442558ee526880aab6cfa"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "3c7794186dd5fedc65dea3a504e06c54"
server_key_iv: "cdee1369082bb66bf41540c13f324272"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.076006 6618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "a5bca2680d9442558ee526880aab6cfa"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "3c7794186dd5fedc65dea3a504e06c54"
server_key_iv: "cdee1369082bb66bf41540c13f324272"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.079977 6618 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:44.082528 6634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.083806 6618 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:44.083956 6618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "a5bca2680d9442558ee526880aab6cfa"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "3c7794186dd5fedc65dea3a504e06c54"
server_key_iv: "cdee1369082bb66bf41540c13f324272"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.084092 6618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:44.105417 6618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:44.108613 6618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:44.108841 6618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:44.109510 6618 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:44.110469 6618 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:44.110539 6618 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.110610 6618 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:44.110648 6618 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.119673 6618 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:41649
I20260811 04:36:44.119724 6754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:41649 every 8 connection(s)
I20260811 04:36:44.120667 6618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:44.130748 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6618
I20260811 04:36:44.130872 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:44.131170 3002 external_mini_cluster.cc:1428] Setting key 165dbe3247ffd4f64ff4898f2eca467e
I20260811 04:36:44.134944 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:44.122564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:50451 (local address 127.2.238.190:42311)
0811 04:36:44.122715 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:44.122719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:44.123558 (+ 839us) server_negotiation.cc:407] Connection header received
0811 04:36:44.124925 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:44.124928 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:44.124979 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:44.125052 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:44.126351 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.126887 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.128045 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.128288 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.128370 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:44.131186 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:44.131190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:44.131191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:44.131214 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:44.132859 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.133366 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.133370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.133372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.133455 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.133838 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.133842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.133844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.134031 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:44.134140 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:44.134652 (+ 512us) server_negotiation.cc:299] Negotiation successful
0811 04:36:44.134781 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:36:44.135944 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:44.122957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42311 (local address 127.2.238.130:50451)
0811 04:36:44.123399 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:44.123443 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:44.124722 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:44.125231 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:44.125237 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:44.125634 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:44.126218 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:44.126233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.127018 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.127021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:44.127915 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:44.127924 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.128556 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.128562 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:44.128764 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:44.129398 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:44.129415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:44.131021 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:44.132970 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:44.132977 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:44.132995 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:44.133265 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:44.133545 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:44.133549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:44.133551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:44.133698 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:44.134140 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:44.134148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:44.134414 (+ 266us) client_negotiation.cc:769] Sending connection context
0811 04:36:44.134613 (+ 199us) client_negotiation.cc:240] Negotiation successful
0811 04:36:44.134978 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":122,"threads_started":1}
I20260811 04:36:44.137252 6755 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42311
I20260811 04:36:44.137513 6755 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:44.138092 6755 heartbeater.cc:511] Master 127.2.238.190:42311 requested a full tablet report, sending...
I20260811 04:36:44.139205 6412 ts_manager.cc:194] Registered new tserver with Master: a5bca2680d9442558ee526880aab6cfa (127.2.238.130:41649)
I20260811 04:36:44.139809 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:50451
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:44.161288 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:44.280874 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:44.281169 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:44.281234 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:44.285357 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:44.285432 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:44.285492 6760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:44.291256 6760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6760
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:44.292510 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:44.293661 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:44.303963 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:44.303972 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:44.303972 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:44.304157 6760 server_base.cc:1034] running on GCE node
I20260811 04:36:44.304751 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:44.305455 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:44.306736 6760 hybrid_clock.cc:648] HybridClock initialized: now 1786423004306695 us; error 51 us; skew 500 ppm
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:44.309623 6760 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:44.310633 6760 webserver.cc:460] Webserver started at http://127.2.238.131:45671/ using document root <none> and password file <none>
I20260811 04:36:44.311208 6760 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:44.311297 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:44.311535 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:44.313619 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "baa653a953594b6faa925880a71f4dfd"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "1d390a1364c152076302609eab7fd68b"
server_key_iv: "71c0dc3bfd4f0525d1283de0bf2d5d28"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.314252 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "baa653a953594b6faa925880a71f4dfd"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "1d390a1364c152076302609eab7fd68b"
server_key_iv: "71c0dc3bfd4f0525d1283de0bf2d5d28"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.318504 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:36:44.321440 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.322808 6760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:36:44.322948 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "baa653a953594b6faa925880a71f4dfd"
format_stamp: "Formatted at 2026-08-11 04:36:44 on dist-test-slave-0kls"
server_key: "1d390a1364c152076302609eab7fd68b"
server_key_iv: "71c0dc3bfd4f0525d1283de0bf2d5d28"
server_key_version: "encryptionkey@0"
I20260811 04:36:44.323062 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:44.336201 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:44.339591 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:44.339830 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:44.340487 6760 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:44.341342 6760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:44.341413 6760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.341485 6760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:44.341534 6760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:44.350966 6760 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:46045
I20260811 04:36:44.350992 6898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:46045 every 8 connection(s)
I20260811 04:36:44.351841 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:44.359817 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6760
I20260811 04:36:44.359954 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:44.360275 3002 external_mini_cluster.cc:1428] Setting key 371320394eeb782d49284ab48155fca1
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:44.366468 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:44.353593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:41599 (local address 127.2.238.190:42311)
0811 04:36:44.353743 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:44.353748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:44.354613 (+ 865us) server_negotiation.cc:407] Connection header received
0811 04:36:44.355964 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:44.355967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:44.356013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:44.356081 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:44.357434 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.358197 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.359342 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.359559 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.359688 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:44.362711 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:44.362715 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:44.362717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:44.362748 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:44.364497 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.364991 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.364994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.364995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.365087 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.365384 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.365386 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.365388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.365556 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:44.365638 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:44.366233 (+ 595us) server_negotiation.cc:299] Negotiation successful
0811 04:36:44.366338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:36:44.367276 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:44.354029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42311 (local address 127.2.238.131:41599)
0811 04:36:44.354465 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:44.354506 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:44.355785 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:44.356409 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:44.356416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:44.356766 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:44.357289 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:44.357303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.358356 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.358359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:44.359234 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:44.359244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.359729 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.359733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:44.359949 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:44.360621 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:44.360644 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:44.362411 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:44.364623 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:44.364627 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:44.364639 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:44.364900 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:44.365168 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:44.365171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:44.365172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:44.365304 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:44.365670 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:44.365676 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:44.365989 (+ 313us) client_negotiation.cc:769] Sending connection context
0811 04:36:44.366156 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 04:36:44.366403 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":109,"threads_started":1}
I20260811 04:36:44.368455 6899 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42311
I20260811 04:36:44.368680 6899 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:44.369127 6899 heartbeater.cc:511] Master 127.2.238.190:42311 requested a full tablet report, sending...
I20260811 04:36:44.370193 6412 ts_manager.cc:194] Registered new tserver with Master: baa653a953594b6faa925880a71f4dfd (127.2.238.131:46045)
I20260811 04:36:44.370721 6412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:41599
I20260811 04:36:44.374444 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423004, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:44.400660 6453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:44.392581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.2.238.190:42311)
0811 04:36:44.392792 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:44.392796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:44.392872 (+ 76us) server_negotiation.cc:407] Connection header received
0811 04:36:44.392994 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:44.392996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:44.393042 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:44.393111 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:44.393829 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.394352 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.395166 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:44.395376 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:44.395452 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:44.397885 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:44.397888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:44.397891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:44.397915 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:44.399134 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.399504 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.399506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.399508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.399583 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:44.399821 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:44.399823 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:44.399824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:44.400040 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:44.400144 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:44.400346 (+ 202us) server_negotiation.cc:299] Negotiation successful
0811 04:36:44.400443 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260811 04:36:44.401110 6412 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:53036
I20260811 04:36:44.402020 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6479
I20260811 04:36:44.410271 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6618
I20260811 04:36:44.418113 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6760
I20260811 04:36:44.428270 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6380
2026-08-11T04:36:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2284 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6912](info): set up 2 sockets
Aug 11 04:36:44 dist-test-slave-0kls krb5kdc[6912](info): commencing operation
krb5kdc: starting...
W20260811 04:36:45.467186 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:45 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:45.597048 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36077
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:45.712760 6927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:45.713058 6927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:45.713099 6927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:45.717274 6927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:45.717330 6927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:45.717350 6927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:45.717402 6927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:45.717433 6927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:45.722731 6927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:45.723897 6927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:45.725028 6927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:45.734234 6935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:45.734244 6932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:45.734295 6927 server_base.cc:1034] running on GCE node
W20260811 04:36:45.734244 6933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:45.735177 6927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:45.736369 6927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:45.737648 6927 hybrid_clock.cc:648] HybridClock initialized: now 1786423005737615 us; error 43 us; skew 500 ppm
Aug 11 04:36:45 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423005, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:45.744773 6927 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:45.745719 6927 webserver.cc:460] Webserver started at http://127.2.238.190:45201/ using document root <none> and password file <none>
I20260811 04:36:45.746342 6927 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:45.746407 6927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:45.746658 6927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:45.748780 6927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "60511c47f0b34f828722c469e90b7899"
format_stamp: "Formatted at 2026-08-11 04:36:45 on dist-test-slave-0kls"
server_key: "1902dc191c12fb2f4d0ced0b2ce15ca4"
server_key_iv: "0dedf17749037b680b4d807edd871d6f"
server_key_version: "encryptionkey@0"
I20260811 04:36:45.749428 6927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "60511c47f0b34f828722c469e90b7899"
format_stamp: "Formatted at 2026-08-11 04:36:45 on dist-test-slave-0kls"
server_key: "1902dc191c12fb2f4d0ced0b2ce15ca4"
server_key_iv: "0dedf17749037b680b4d807edd871d6f"
server_key_version: "encryptionkey@0"
I20260811 04:36:45.753263 6927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:45.755709 6942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:45.756996 6927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:45.757146 6927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "60511c47f0b34f828722c469e90b7899"
format_stamp: "Formatted at 2026-08-11 04:36:45 on dist-test-slave-0kls"
server_key: "1902dc191c12fb2f4d0ced0b2ce15ca4"
server_key_iv: "0dedf17749037b680b4d807edd871d6f"
server_key_version: "encryptionkey@0"
I20260811 04:36:45.757262 6927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:45.788291 6927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:45.791522 6927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:45.791770 6927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:45.799338 6927 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:36077
I20260811 04:36:45.799407 7000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:36077 every 8 connection(s)
I20260811 04:36:45.800402 6927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:45.803432 7001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:45.804929 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 6927
I20260811 04:36:45.805052 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:45.805392 3002 external_mini_cluster.cc:1428] Setting key 3328f6333638d1056726c72106cb768e
I20260811 04:36:45.810287 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899: Bootstrap starting.
I20260811 04:36:45.813030 7001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:45.814134 7001 log.cc:824] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:36:45 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:45.816606 7001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899: No bootstrap required, opened a new log
I20260811 04:36:45.820748 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:45.806911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39490 (local address 127.2.238.190:36077)
0811 04:36:45.807366 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:45.807382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:45.807418 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:36:45.808021 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:45.808038 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:45.808390 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:45.808677 (+ 287us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:45.809660 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:45.810794 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:45.811967 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:45.812444 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:45.812569 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:45.815484 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:45.815490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:45.815524 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:45.815556 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:45.817482 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:45.817974 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:45.817980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:45.817987 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:45.818132 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:45.818389 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:45.818392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:45.818394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:45.818936 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:45.819181 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:45.819403 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 04:36:45.819717 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":165,"threads_started":1}
I20260811 04:36:45.821431 7001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } }
I20260811 04:36:45.821652 7001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:45.821714 7001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60511c47f0b34f828722c469e90b7899, State: Initialized, Role: FOLLOWER
I20260811 04:36:45.822247 7001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } }
I20260811 04:36:45.822409 7001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:45.822484 7001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:45.822587 7001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:45.823568 7001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } }
I20260811 04:36:45.823937 7001 leader_election.cc:302] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60511c47f0b34f828722c469e90b7899; no voters:
I20260811 04:36:45.824270 7001 leader_election.cc:288] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:45.824451 7006 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:45.824698 7006 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [term 1 LEADER]: Becoming Leader. State: Replica: 60511c47f0b34f828722c469e90b7899, State: Running, Role: LEADER
I20260811 04:36:45.825053 7006 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } }
I20260811 04:36:45.825584 7001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:45.827057 7008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60511c47f0b34f828722c469e90b7899. Latest consensus state: current_term: 1 leader_uuid: "60511c47f0b34f828722c469e90b7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } } }
I20260811 04:36:45.827291 7008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:45.827545 7007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60511c47f0b34f828722c469e90b7899" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60511c47f0b34f828722c469e90b7899" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36077 } } }
I20260811 04:36:45.827698 7007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:45.827744 7015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:45.830972 7015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:45.837014 7015 catalog_manager.cc:1347] Generated new cluster ID: 8775a0f2470d414cb4a1d769441aef8c
I20260811 04:36:45.837090 7015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:45.847299 7015 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:45.848191 7015 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:45.852751 7015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60511c47f0b34f828722c469e90b7899: Generated new TSK 0
I20260811 04:36:45.853425 7015 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:45.888723 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:46.040392 7027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:46.040834 7027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:46.040920 7027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:46.045246 7027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:46.045348 7027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:46.045454 7027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:46.051317 7027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:46.052618 7027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:46.053735 7027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:46.063936 7033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:46.063941 7035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:46.063941 7032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:46.064350 7027 server_base.cc:1034] running on GCE node
I20260811 04:36:46.064880 7027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:46.065557 7027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:46.066758 7027 hybrid_clock.cc:648] HybridClock initialized: now 1786423006066715 us; error 55 us; skew 500 ppm
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:46.069515 7027 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:46.070437 7027 webserver.cc:460] Webserver started at http://127.2.238.129:35459/ using document root <none> and password file <none>
I20260811 04:36:46.071024 7027 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:46.071120 7027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:46.071365 7027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:46.073449 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "faf75f50a33349db98c90486d6995492"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "ba3d9f66a95db315b8be1335102ba37a"
server_key_iv: "16074220fb7f6400d125e0edecd131e1"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.074080 7027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "faf75f50a33349db98c90486d6995492"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "ba3d9f66a95db315b8be1335102ba37a"
server_key_iv: "16074220fb7f6400d125e0edecd131e1"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.077931 7027 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:36:46.080425 7042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.081622 7027 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:46.081775 7027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "faf75f50a33349db98c90486d6995492"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "ba3d9f66a95db315b8be1335102ba37a"
server_key_iv: "16074220fb7f6400d125e0edecd131e1"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.081944 7027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:46.095151 7027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:46.098006 7027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:46.098219 7027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:46.098832 7027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:46.099679 7027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:46.099758 7027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.099869 7027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:46.099922 7027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.109066 7027 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44239
I20260811 04:36:46.109087 7162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44239 every 8 connection(s)
I20260811 04:36:46.110070 7027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:46.116746 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7027
I20260811 04:36:46.116912 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:46.117230 3002 external_mini_cluster.cc:1428] Setting key 9017b54c8377993f9294391f3a018950
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:46.124599 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.111928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:39983 (local address 127.2.238.190:36077)
0811 04:36:46.112066 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:46.112075 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:46.112895 (+ 820us) server_negotiation.cc:407] Connection header received
0811 04:36:46.114314 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:46.114317 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:46.114367 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:46.114444 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:46.115579 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.116117 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.117341 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.117584 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.117698 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.120653 (+ 2955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:46.120659 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:46.120661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:46.120697 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:46.122284 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.122927 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.122929 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.122930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.123010 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.123404 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.123406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.123407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.123580 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:46.123656 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:46.124363 (+ 707us) server_negotiation.cc:299] Negotiation successful
0811 04:36:46.124474 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:46.125618 7165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.112265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36077 (local address 127.2.238.129:39983)
0811 04:36:46.112727 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:46.112773 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:46.114107 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:46.114607 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:46.114613 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:46.114960 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:46.115446 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.115460 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.116223 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.116226 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.117191 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.117201 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.117747 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.117752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.117960 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.118690 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:46.118711 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:46.120472 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:46.122404 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.122413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.122433 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.122824 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.123102 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.123107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.123112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.123309 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.123698 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:46.123708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:46.124076 (+ 368us) client_negotiation.cc:769] Sending connection context
0811 04:36:46.124322 (+ 246us) client_negotiation.cc:240] Negotiation successful
0811 04:36:46.124725 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":116,"threads_started":1}
I20260811 04:36:46.127014 7163 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36077
I20260811 04:36:46.127270 7163 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:46.127871 7163 heartbeater.cc:511] Master 127.2.238.190:36077 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:46.129455 6960 ts_manager.cc:194] Registered new tserver with Master: faf75f50a33349db98c90486d6995492 (127.2.238.129:44239)
I20260811 04:36:46.130903 6960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:39983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:46.147204 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:46.263646 7168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:46.263933 7168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:46.264021 7168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:46.268200 7168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:46.268289 7168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:46.268376 7168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:46.273885 7168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:46.275005 7168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:46.276046 7168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:46.285713 7173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:46.285753 7176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:46.285707 7174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:46.285966 7168 server_base.cc:1034] running on GCE node
I20260811 04:36:46.286417 7168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:46.286999 7168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:46.288162 7168 hybrid_clock.cc:648] HybridClock initialized: now 1786423006288147 us; error 25 us; skew 500 ppm
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:46.291054 7168 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:46.291935 7168 webserver.cc:460] Webserver started at http://127.2.238.130:43655/ using document root <none> and password file <none>
I20260811 04:36:46.292480 7168 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:46.292539 7168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:46.292722 7168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:46.294845 7168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "c06545fdaa6d496baca54ac45d667592"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "04d92236872610b28228cfcd549a4e42"
server_key_iv: "5faa16b029f1269d4c98f16405156edd"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.295382 7168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "c06545fdaa6d496baca54ac45d667592"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "04d92236872610b28228cfcd549a4e42"
server_key_iv: "5faa16b029f1269d4c98f16405156edd"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.299170 7168 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:46.301549 7183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.302789 7168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:46.302909 7168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "c06545fdaa6d496baca54ac45d667592"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "04d92236872610b28228cfcd549a4e42"
server_key_iv: "5faa16b029f1269d4c98f16405156edd"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.303013 7168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:46.323685 7168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:46.326622 7168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:46.326826 7168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:46.327497 7168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:46.328428 7168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:46.328472 7168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.328531 7168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:46.328564 7168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.337743 7168 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:41887
I20260811 04:36:46.337765 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:41887 every 8 connection(s)
I20260811 04:36:46.338717 7168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:46.344638 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7168
I20260811 04:36:46.344774 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:46.345052 3002 external_mini_cluster.cc:1428] Setting key 2ef3081cad0c3a98a802e5e77eb06468
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:46.353466 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.340593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41093 (local address 127.2.238.190:36077)
0811 04:36:46.340729 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:46.340734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:46.341573 (+ 839us) server_negotiation.cc:407] Connection header received
0811 04:36:46.343092 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:46.343096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:46.343158 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:46.343274 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:46.344497 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.345096 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.346635 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.346869 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.346979 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.349720 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:46.349723 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:46.349725 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:46.349748 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:46.351148 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.351662 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.351665 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.351666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.351785 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.352114 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.352118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.352120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.352308 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:46.352411 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:46.353044 (+ 633us) server_negotiation.cc:299] Negotiation successful
0811 04:36:46.353197 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:36:46.354331 7306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.340985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36077 (local address 127.2.238.130:41093)
0811 04:36:46.341428 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:46.341470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:46.342874 (+ 1404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:46.343455 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:46.343462 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:46.343826 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:46.344318 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.344333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.345268 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.345271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.346514 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.346524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.347059 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.347065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.347289 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.347907 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:46.347933 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:46.349548 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:46.351263 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.351269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.351285 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.351539 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.351885 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.351888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.351890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.352024 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.352409 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:46.352416 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:46.352724 (+ 308us) client_negotiation.cc:769] Sending connection context
0811 04:36:46.352969 (+ 245us) client_negotiation.cc:240] Negotiation successful
0811 04:36:46.353268 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
I20260811 04:36:46.355590 7304 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36077
I20260811 04:36:46.355863 7304 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:36:46.356487 7304 heartbeater.cc:511] Master 127.2.238.190:36077 requested a full tablet report, sending...
I20260811 04:36:46.357555 6960 ts_manager.cc:194] Registered new tserver with Master: c06545fdaa6d496baca54ac45d667592 (127.2.238.130:41887)
I20260811 04:36:46.358242 6960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:41093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:46.374831 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:46003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:46.487814 7309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:46.488070 7309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:46.488111 7309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:46.492089 7309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:46.492159 7309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:46.492214 7309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:46.497264 7309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:46003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:46.498346 7309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:46.499348 7309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:46.509397 7314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:46.509428 7309 server_base.cc:1034] running on GCE node
W20260811 04:36:46.509351 7315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:46.509547 7317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:46.510123 7309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:46.510771 7309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:46.511960 7309 hybrid_clock.cc:648] HybridClock initialized: now 1786423006511945 us; error 25 us; skew 500 ppm
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:46.514719 7309 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:46.515606 7309 webserver.cc:460] Webserver started at http://127.2.238.131:32881/ using document root <none> and password file <none>
I20260811 04:36:46.516170 7309 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:46.516266 7309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:46.516498 7309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:46.518524 7309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "9de50c824c3c4d829418897cc7351fac"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "6334a6004dbb2a050a33ce67fcd974b2"
server_key_iv: "c9d12bc8f1228398cb7db7e8ecaff542"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.519073 7309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "9de50c824c3c4d829418897cc7351fac"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "6334a6004dbb2a050a33ce67fcd974b2"
server_key_iv: "c9d12bc8f1228398cb7db7e8ecaff542"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.522750 7309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:36:46.525100 7324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.526472 7309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:36:46.526643 7309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "9de50c824c3c4d829418897cc7351fac"
format_stamp: "Formatted at 2026-08-11 04:36:46 on dist-test-slave-0kls"
server_key: "6334a6004dbb2a050a33ce67fcd974b2"
server_key_iv: "c9d12bc8f1228398cb7db7e8ecaff542"
server_key_version: "encryptionkey@0"
I20260811 04:36:46.526764 7309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:46.547565 7309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:46.550899 7309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:46.551049 7309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:46.551672 7309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:46.552471 7309 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:46.552551 7309 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.552609 7309 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:46.552632 7309 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:46.561017 7309 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:39187
I20260811 04:36:46.561058 7443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:39187 every 8 connection(s)
I20260811 04:36:46.561991 7309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:46.571872 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7309
I20260811 04:36:46.572036 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:46.572322 3002 external_mini_cluster.cc:1428] Setting key 491e8c2a6791002f2019e44dd6f35e98
I20260811 04:36:46.575362 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.563786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:58291 (local address 127.2.238.190:36077)
0811 04:36:46.563927 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:46.563931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:46.564685 (+ 754us) server_negotiation.cc:407] Connection header received
0811 04:36:46.566013 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:46.566016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:46.566067 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:46.566141 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:46.567221 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.567727 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.568853 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.569069 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.569143 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.571701 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:46.571704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:46.571705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:46.571729 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:46.573185 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.573743 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.573747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.573749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.573866 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.574276 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.574278 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.574279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.574477 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:46.574582 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:46.575111 (+ 529us) server_negotiation.cc:299] Negotiation successful
0811 04:36:46.575206 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:36:46.576162 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.564084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36077 (local address 127.2.238.131:58291)
0811 04:36:46.564522 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:46.564564 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:46.565804 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:46.566272 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:46.566278 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:46.566618 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:46.567095 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.567108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.567845 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.567848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.568736 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:46.568745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.569205 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.569208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:46.569347 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.569969 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:46.569986 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:46.571525 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:46.573277 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.573283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.573298 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.573617 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.573980 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:46.573983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:46.573984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:46.574162 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:46.574600 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:46.574608 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:46.574887 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 04:36:46.575060 (+ 173us) client_negotiation.cc:240] Negotiation successful
0811 04:36:46.575341 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":117,"threads_started":1}
I20260811 04:36:46.577348 7444 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36077
I20260811 04:36:46.577556 7444 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:46.578109 7444 heartbeater.cc:511] Master 127.2.238.190:36077 requested a full tablet report, sending...
I20260811 04:36:46.579126 6960 ts_manager.cc:194] Registered new tserver with Master: 9de50c824c3c4d829418897cc7351fac (127.2.238.131:39187)
I20260811 04:36:46.579660 6960 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:58291
I20260811 04:36:46.586592 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:46.613453 7453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.604115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.2.238.129:44239)
0811 04:36:46.604403 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:46.604422 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:46.604463 (+ 41us) server_negotiation.cc:407] Connection header received
0811 04:36:46.604527 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:46.604532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:46.604744 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:46.604893 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:46.605573 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.606464 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.607314 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.607731 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.607861 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.610250 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:46.610260 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:46.610283 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:46.610309 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:46.611819 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.612208 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.612212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.612214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.612301 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.612530 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.612532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.612533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.612771 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:46.612935 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:46.613071 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 04:36:46.613237 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":69,"threads_started":1}
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[6912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:46.624156 7004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:46.616361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39504 (local address 127.2.238.190:36077)
0811 04:36:46.616498 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:46.616504 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:46.616519 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:46.616630 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:46.616634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:46.616680 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:46.616762 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:46.617335 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.617930 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.618704 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:46.618924 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:46.619031 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:46.621298 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:46.621301 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:46.621303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:46.621327 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:46.622775 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.623127 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.623129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.623131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.623204 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:46.623436 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:46.623439 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:46.623440 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:46.623646 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:46.623757 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:46.623952 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 04:36:46.624040 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 04:36:46.624620 6960 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39504
I20260811 04:36:46.625365 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7027
I20260811 04:36:46.634407 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7168
I20260811 04:36:46.641601 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7309
I20260811 04:36:46.649417 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 6927
2026-08-11T04:36:46Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2220 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[7458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[7458](info): set up 2 sockets
Aug 11 04:36:46 dist-test-slave-0kls krb5kdc[7458](info): commencing operation
krb5kdc: starting...
W20260811 04:36:47.736688 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:47 dist-test-slave-0kls krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:47.870559 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40635
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:40635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:47.988477 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:47.988781 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:47.988827 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:47.992882 7473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:47.992960 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:47.992980 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:47.992996 7473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:47.993011 7473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:47.998554 7473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:40635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:47.999667 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:48.000705 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:48.010128 7479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.010671 7482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:48.011170 7473 server_base.cc:1034] running on GCE node
W20260811 04:36:48.011322 7480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:48.012166 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:48.013643 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:48.014847 7473 hybrid_clock.cc:648] HybridClock initialized: now 1786423008014832 us; error 27 us; skew 500 ppm
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:48.017655 7473 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:48.018512 7473 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:48.018606 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:48.018849 7473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:48.021027 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "c6bd35a367cb46319cf22bca02bcf467"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "e5ebb65ce9b84370cd50a2e86e71fb4d"
server_key_iv: "cfa0a5c6294f21b42d0e3cdbc9bd6c0b"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.021623 7473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "c6bd35a367cb46319cf22bca02bcf467"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "e5ebb65ce9b84370cd50a2e86e71fb4d"
server_key_iv: "cfa0a5c6294f21b42d0e3cdbc9bd6c0b"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.025754 7473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 04:36:48.028407 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.029704 7473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:48.029891 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "c6bd35a367cb46319cf22bca02bcf467"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "e5ebb65ce9b84370cd50a2e86e71fb4d"
server_key_iv: "cfa0a5c6294f21b42d0e3cdbc9bd6c0b"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.029989 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:48.040356 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:48.043314 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:48.043490 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:48.051848 7473 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:40635
I20260811 04:36:48.052119 7543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:40635 every 8 connection(s)
I20260811 04:36:48.052582 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:48.055565 7544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:48.059243 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7473
I20260811 04:36:48.059358 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:48.059679 3002 external_mini_cluster.cc:1428] Setting key cfc19c76c392695ae77a88c2445bd167
I20260811 04:36:48.062382 7544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: Bootstrap starting.
I20260811 04:36:48.065270 7544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:48.066218 7544 log.cc:824] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:48.068660 7544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: No bootstrap required, opened a new log
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:48.071935 7544 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } }
I20260811 04:36:48.072201 7544 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:48.072295 7544 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bd35a367cb46319cf22bca02bcf467, State: Initialized, Role: FOLLOWER
I20260811 04:36:48.072804 7544 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } }
I20260811 04:36:48.072963 7544 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:48.073051 7544 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:48.073153 7544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:48.074198 7544 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } }
I20260811 04:36:48.074558 7544 leader_election.cc:302] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6bd35a367cb46319cf22bca02bcf467; no voters:
I20260811 04:36:48.074946 7544 leader_election.cc:288] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:48.075030 7549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:48.075297 7549 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [term 1 LEADER]: Becoming Leader. State: Replica: c6bd35a367cb46319cf22bca02bcf467, State: Running, Role: LEADER
I20260811 04:36:48.075670 7549 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } }
I20260811 04:36:48.076375 7544 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:48.076488 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.061261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42238 (local address 127.2.238.190:40635)
0811 04:36:48.061742 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.061763 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.061843 (+ 80us) server_negotiation.cc:407] Connection header received
0811 04:36:48.062717 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.062742 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.063196 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.063569 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:48.064280 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.065773 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.066761 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.067365 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.067491 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.070345 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:48.070351 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:48.070367 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:48.070396 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:48.072894 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.073539 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.073544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.073550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.073670 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.074027 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.074031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.074034 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.074528 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:48.074751 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.075056 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.075580 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":132,"threads_started":1}
I20260811 04:36:48.078469 7552 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6bd35a367cb46319cf22bca02bcf467. Latest consensus state: current_term: 1 leader_uuid: "c6bd35a367cb46319cf22bca02bcf467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } } }
I20260811 04:36:48.078619 7552 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:48.079267 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6bd35a367cb46319cf22bca02bcf467" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6bd35a367cb46319cf22bca02bcf467" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40635 } } }
I20260811 04:36:48.079419 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467 [sys.catalog]: This master's current role is: LEADER
W20260811 04:36:48.081496 7565 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:36:48.081580 7565 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:36:48.081672 7566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:48.082643 7566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:48.088438 7566 catalog_manager.cc:1347] Generated new cluster ID: a86129773dbc4803971ae39338e927a2
I20260811 04:36:48.088526 7566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:48.100838 7566 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:48.102239 7566 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:48.111061 7566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: Generated new TSK 0
I20260811 04:36:48.111825 7566 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:48.144843 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:48.267561 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:48.267906 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:48.267994 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:48.272146 7572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:48.272260 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:48.272359 7572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:48.278450 7572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:48.279700 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:48.280943 7572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:48.290840 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.290804 7578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.290822 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:48.291139 7572 server_base.cc:1034] running on GCE node
I20260811 04:36:48.291666 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:48.292398 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:48.293617 7572 hybrid_clock.cc:648] HybridClock initialized: now 1786423008293599 us; error 29 us; skew 500 ppm
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:48.296748 7572 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:48.297533 7572 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:48.297602 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:48.297837 7572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:48.300019 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "c0d3139af5344a29acd379a0996ee754"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "dcacb940f251a1d3c3117cb467ae4c4f"
server_key_iv: "e7243a96c165dee21790fe637528e25d"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.300572 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "c0d3139af5344a29acd379a0996ee754"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "dcacb940f251a1d3c3117cb467ae4c4f"
server_key_iv: "e7243a96c165dee21790fe637528e25d"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.304737 7572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260811 04:36:48.307510 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.308876 7572 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:48.308993 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "c0d3139af5344a29acd379a0996ee754"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "dcacb940f251a1d3c3117cb467ae4c4f"
server_key_iv: "e7243a96c165dee21790fe637528e25d"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.309098 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:48.330856 7572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:48.334755 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:48.334985 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:48.335711 7572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:48.336669 7572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:48.336719 7572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.336759 7572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:48.336782 7572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.346019 7572 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40239
I20260811 04:36:48.346045 7707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40239 every 8 connection(s)
I20260811 04:36:48.346690 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:48.353653 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7572
I20260811 04:36:48.353766 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:48.354115 3002 external_mini_cluster.cc:1428] Setting key f686936ad87b8bf9e93b569e4d846665
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:48.361045 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.348655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:46819 (local address 127.2.238.190:40635)
0811 04:36:48.348842 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.348847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.349765 (+ 918us) server_negotiation.cc:407] Connection header received
0811 04:36:48.351161 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.351164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.351215 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.351297 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:48.352532 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.353077 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.354479 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.354801 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.354932 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.357444 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:48.357447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:48.357450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:48.357477 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:48.358865 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.359420 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.359425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.359427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.359512 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.359887 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.359890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.359892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.360088 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:48.360202 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.360727 (+ 525us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.360869 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260811 04:36:48.362017 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.349089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40635 (local address 127.2.238.129:46819)
0811 04:36:48.349599 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.349647 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.350954 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.351473 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.351481 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.351872 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:48.352392 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.352406 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.353191 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.353193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.354321 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.354332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.354982 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.354987 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.355173 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.355790 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:48.355807 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:48.357265 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:48.358987 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.358993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.359011 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.359287 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.359610 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.359614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.359616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.359771 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.360203 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:48.360211 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:48.360485 (+ 274us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.360687 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.361049 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260811 04:36:48.363448 7708 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40635
I20260811 04:36:48.363744 7708 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:48.364383 7708 heartbeater.cc:511] Master 127.2.238.190:40635 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:48.366168 7502 ts_manager.cc:194] Registered new tserver with Master: c0d3139af5344a29acd379a0996ee754 (127.2.238.129:40239)
I20260811 04:36:48.367470 7502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:46819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:48.384833 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:48.502166 7715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:48.502439 7715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:48.502482 7715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:48.506537 7715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:48.506608 7715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:48.506669 7715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:48.511974 7715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:48.513170 7715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:48.514349 7715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:48.523890 7725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.523892 7722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.523892 7721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:48.524134 7715 server_base.cc:1034] running on GCE node
I20260811 04:36:48.524642 7715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:48.525317 7715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:48.526499 7715 hybrid_clock.cc:648] HybridClock initialized: now 1786423008526483 us; error 27 us; skew 500 ppm
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:48.529356 7715 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:48.530194 7715 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:48.530299 7715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:48.530546 7715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:48.532604 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "cb9caa4f155445528f2562bddbe83f98"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "c7157ebfd737dddee73b7ca525e7ba57"
server_key_iv: "80190df0b5984a76a579f5cb041b61fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.533241 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "cb9caa4f155445528f2562bddbe83f98"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "c7157ebfd737dddee73b7ca525e7ba57"
server_key_iv: "80190df0b5984a76a579f5cb041b61fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.537114 7715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:48.539703 7731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.540963 7715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:36:48.541126 7715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "cb9caa4f155445528f2562bddbe83f98"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "c7157ebfd737dddee73b7ca525e7ba57"
server_key_iv: "80190df0b5984a76a579f5cb041b61fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.541267 7715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:48.553987 7715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:48.557655 7715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:48.557947 7715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:48.558674 7715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:48.559746 7715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:48.559851 7715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.559937 7715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:48.559979 7715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.570043 7715 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43593
I20260811 04:36:48.570062 7852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43593 every 8 connection(s)
I20260811 04:36:48.570703 7715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:48.572553 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7715
I20260811 04:36:48.572682 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:48.573014 3002 external_mini_cluster.cc:1428] Setting key ed3f5495fd1df7f4cd11568f0fcd907d
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:36:48.587128 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.572523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:42235 (local address 127.2.238.190:40635)
0811 04:36:48.572668 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.572673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.573872 (+ 1199us) server_negotiation.cc:407] Connection header received
0811 04:36:48.575343 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.575347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.575403 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.575482 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:48.576819 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.577426 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.579429 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.579729 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.579860 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.583227 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:48.583238 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:48.583240 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:48.583268 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:48.584710 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.585415 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.585418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.585420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.585503 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.585925 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.585928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.585929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.586187 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:48.586280 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.586884 (+ 604us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.587000 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
I20260811 04:36:48.588183 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.573075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40635 (local address 127.2.238.130:42235)
0811 04:36:48.573668 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.573724 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.575126 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.575666 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.575674 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.576061 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:48.576670 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.576687 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.577606 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.577612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.579287 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.579308 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.579928 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.579935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.580201 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.580914 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:48.580939 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:48.582989 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:48.584853 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.584861 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.584881 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.585298 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.585614 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.585617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.585618 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.585778 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.586317 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:48.586328 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:48.586627 (+ 299us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.586820 (+ 193us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.587111 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":141,"threads_started":1}
I20260811 04:36:48.589511 7853 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40635
I20260811 04:36:48.589839 7853 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:48.590448 7853 heartbeater.cc:511] Master 127.2.238.190:40635 requested a full tablet report, sending...
I20260811 04:36:48.591702 7502 ts_manager.cc:194] Registered new tserver with Master: cb9caa4f155445528f2562bddbe83f98 (127.2.238.130:43593)
I20260811 04:36:48.592563 7502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:42235
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:48.605362 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33723
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:48.727897 7858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:48.728209 7858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:48.728255 7858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:48.732349 7858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:48.732434 7858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:48.732496 7858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:48.738523 7858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33723
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:48.739845 7858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:48.740994 7858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:48.751535 7864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.751565 7866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:48.751536 7863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:48.751840 7858 server_base.cc:1034] running on GCE node
I20260811 04:36:48.752359 7858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:48.753033 7858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:48.754230 7858 hybrid_clock.cc:648] HybridClock initialized: now 1786423008754213 us; error 28 us; skew 500 ppm
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:48.757113 7858 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:48.758049 7858 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:48.758153 7858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:48.758410 7858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:48.760650 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "e210373bd12a4b32b533e6e8b120c92e"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "a58e45f38e4bcc5c1758b11b0f07b1b9"
server_key_iv: "3f8e62c2194e4a4b8797564447c4c5a3"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.761288 7858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "e210373bd12a4b32b533e6e8b120c92e"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "a58e45f38e4bcc5c1758b11b0f07b1b9"
server_key_iv: "3f8e62c2194e4a4b8797564447c4c5a3"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.765417 7858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:48.768285 7874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.769641 7858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:48.769876 7858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "e210373bd12a4b32b533e6e8b120c92e"
format_stamp: "Formatted at 2026-08-11 04:36:48 on dist-test-slave-0kls"
server_key: "a58e45f38e4bcc5c1758b11b0f07b1b9"
server_key_iv: "3f8e62c2194e4a4b8797564447c4c5a3"
server_key_version: "encryptionkey@0"
I20260811 04:36:48.770025 7858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:48.786862 7858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:48.790236 7858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:48.790462 7858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:48.791121 7858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:48.792053 7858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:48.792132 7858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.792210 7858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:48.792264 7858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:48.802179 7858 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:38571
I20260811 04:36:48.802209 7993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:38571 every 8 connection(s)
I20260811 04:36:48.802848 7858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:48.803929 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 7858
I20260811 04:36:48.804054 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:48.804420 3002 external_mini_cluster.cc:1428] Setting key 8fa46fd9a461e6763d729b31252d9b93
Aug 11 04:36:48 dist-test-slave-0kls krb5kdc[7458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423008, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:48.817540 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.804956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:54713 (local address 127.2.238.190:40635)
0811 04:36:48.805119 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.805124 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.806226 (+ 1102us) server_negotiation.cc:407] Connection header received
0811 04:36:48.807852 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.807855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.807905 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.807976 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:48.809227 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.809761 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.810956 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.811175 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.811256 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.813886 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:48.813889 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:48.813891 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:48.813919 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:48.815414 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.815896 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.815899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.815900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.815978 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.816296 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.816298 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.816299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.816487 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:48.816564 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.817182 (+ 618us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.817350 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:36:48.818701 7996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.805433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40635 (local address 127.2.238.131:54713)
0811 04:36:48.806053 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.806110 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.807614 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.808166 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.808174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.808566 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:48.809095 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.809109 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.809918 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.809921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.810837 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.810847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.811327 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.811330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.811480 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.812048 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:48.812066 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:48.813671 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:48.815519 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.815524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.815538 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.815808 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.816058 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:48.816060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:48.816061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:48.816215 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:48.816578 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:48.816583 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:48.816882 (+ 299us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.817130 (+ 248us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.817506 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":164,"threads_started":1}
I20260811 04:36:48.819777 7994 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40635
I20260811 04:36:48.820056 7994 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:48.820556 7994 heartbeater.cc:511] Master 127.2.238.190:40635 requested a full tablet report, sending...
I20260811 04:36:48.821700 7502 ts_manager.cc:194] Registered new tserver with Master: e210373bd12a4b32b533e6e8b120c92e (127.2.238.131:38571)
I20260811 04:36:48.822321 7502 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:54713
I20260811 04:36:48.831015 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:36:48.841161 7547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.833115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.2.238.190:40635)
0811 04:36:48.833262 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.833267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.833486 (+ 219us) server_negotiation.cc:407] Connection header received
0811 04:36:48.833641 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.833645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.833716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.833826 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:48.834632 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.835169 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.836036 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.836275 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.836379 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.837881 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:48.837886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:48.837888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:48.837913 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:48.839352 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.839810 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.839815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.839817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.839903 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:48.840195 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:48.840199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:48.840201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:48.840460 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:48.840601 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.840857 (+ 256us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.840983 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:36:48.844390 7502 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:48.848057 7502 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:48.863761 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.2.238.130:43593)
0811 04:36:48.858639 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.858650 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.858673 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:36:48.858741 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.858746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.858931 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.859083 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:48.859507 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.860310 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.861870 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.862715 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.862857 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.863019 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.863464 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.863581 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":105,"threads_started":1}
I20260811 04:36:48.863755 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43593 (local address 127.0.0.1:50880)
0811 04:36:48.858259 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.858272 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.858349 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.859095 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.859103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.859151 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:48.859364 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.859371 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.860446 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.860452 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.861706 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.861716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.863083 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.863087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.863251 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.863290 (+ 39us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.863423 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.863590 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":127,"spinlock_wait_cycles":11392,"thread_start_us":98,"threads_started":1}
I20260811 04:36:48.864440 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40239 (local address 127.0.0.1:58648)
0811 04:36:48.858054 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.858107 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.858339 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.859111 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.859113 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.859151 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:48.859350 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.859356 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.861857 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.861860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.862954 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.862965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.864051 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.864055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.864145 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.864168 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.864248 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.864315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":7,"spinlock_wait_cycles":5376,"thread_start_us":78,"threads_started":1}
I20260811 04:36:48.864516 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58648 (local address 127.2.238.129:40239)
0811 04:36:48.857526 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.857534 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.858202 (+ 668us) server_negotiation.cc:407] Connection header received
0811 04:36:48.858533 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.858540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.858759 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.858886 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:48.859473 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.860726 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.863081 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.863931 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.864041 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.864167 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.864298 (+ 131us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.864378 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260811 04:36:48.864470 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36202 (local address 127.2.238.131:38571)
0811 04:36:48.858721 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.858759 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.858784 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:36:48.858864 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.858874 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.859050 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.859200 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:48.859598 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.860742 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.862182 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.862726 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.862873 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.863049 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.864178 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.864321 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":95,"threads_started":1}
I20260811 04:36:48.863762 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.857816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:38571 (local address 127.0.0.1:36202)
0811 04:36:48.858375 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.858391 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.858472 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.859359 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.859362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.859380 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:48.859475 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.859482 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.861018 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.861021 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.862066 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.862074 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.863112 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.863115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.863298 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.863323 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.863398 (+ 75us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.863576 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":75,"thread_start_us":101,"threads_started":1}
I20260811 04:36:48.867100 7783 tablet_service.cc:1467] Processing CreateTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a (DEFAULT_TABLE table=test-table [id=db8a21ec28554a8fa5751d6d6ca6a0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:48.867601 7639 tablet_service.cc:1467] Processing CreateTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a (DEFAULT_TABLE table=test-table [id=db8a21ec28554a8fa5751d6d6ca6a0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:48.868115 7783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e42055aaf2844fbb022ceeb7c58871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:48.868716 7639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e42055aaf2844fbb022ceeb7c58871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:48.868477 7925 tablet_service.cc:1467] Processing CreateTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a (DEFAULT_TABLE table=test-table [id=db8a21ec28554a8fa5751d6d6ca6a0f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:48.869619 7925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e42055aaf2844fbb022ceeb7c58871a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:48.875553 8012 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Bootstrap starting.
I20260811 04:36:48.876514 8013 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: Bootstrap starting.
I20260811 04:36:48.877861 8014 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Bootstrap starting.
I20260811 04:36:48.878460 8012 tablet_bootstrap.cc:654] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:48.879079 8013 tablet_bootstrap.cc:654] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:48.879873 8014 tablet_bootstrap.cc:654] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:48.880093 8012 log.cc:824] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:48.880093 8013 log.cc:824] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:48.881232 8014 log.cc:824] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:48.883020 8013 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: No bootstrap required, opened a new log
I20260811 04:36:48.883029 8014 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: No bootstrap required, opened a new log
I20260811 04:36:48.883249 8013 ts_tablet_manager.cc:1397] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:36:48.883248 8014 ts_tablet_manager.cc:1397] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:36:48.883661 8012 tablet_bootstrap.cc:492] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: No bootstrap required, opened a new log
I20260811 04:36:48.883857 8012 ts_tablet_manager.cc:1397] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260811 04:36:48.887019 8013 raft_consensus.cc:348] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.887416 8013 raft_consensus.cc:374] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:48.887496 8013 raft_consensus.cc:729] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d3139af5344a29acd379a0996ee754, State: Initialized, Role: FOLLOWER
I20260811 04:36:48.887439 8014 raft_consensus.cc:348] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.887714 8014 raft_consensus.cc:374] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:48.887784 8014 raft_consensus.cc:729] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e210373bd12a4b32b533e6e8b120c92e, State: Initialized, Role: FOLLOWER
I20260811 04:36:48.887936 8012 raft_consensus.cc:348] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.888052 8013 consensus_queue.cc:260] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.888198 8012 raft_consensus.cc:374] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:48.888262 8012 raft_consensus.cc:729] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb9caa4f155445528f2562bddbe83f98, State: Initialized, Role: FOLLOWER
I20260811 04:36:48.888275 8014 consensus_queue.cc:260] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.888691 8012 consensus_queue.cc:260] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.889181 7708 heartbeater.cc:503] Master 127.2.238.190:40635 was elected leader, sending a full tablet report...
I20260811 04:36:48.889349 8013 ts_tablet_manager.cc:1428] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:36:48.889541 7994 heartbeater.cc:503] Master 127.2.238.190:40635 was elected leader, sending a full tablet report...
I20260811 04:36:48.890496 7853 heartbeater.cc:503] Master 127.2.238.190:40635 was elected leader, sending a full tablet report...
I20260811 04:36:48.890774 8012 ts_tablet_manager.cc:1428] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260811 04:36:48.891034 8014 ts_tablet_manager.cc:1428] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260811 04:36:48.911415 8020 raft_consensus.cc:482] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:48.911662 8020 raft_consensus.cc:504] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.913012 8020 leader_election.cc:288] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0d3139af5344a29acd379a0996ee754 (127.2.238.129:40239), e210373bd12a4b32b533e6e8b120c92e (127.2.238.131:38571)
I20260811 04:36:48.917233 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.913430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:49067 (local address 127.2.238.129:40239)
0811 04:36:48.913607 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.913614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.913636 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:36:48.913702 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.913708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.913778 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.913908 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:48.914334 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.914990 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916167 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.916608 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916706 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.916795 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.917014 (+ 219us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.917093 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:48.917244 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.913343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40239 (local address 127.2.238.130:49067)
0811 04:36:48.913491 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.913508 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.913611 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.914010 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.914013 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.914038 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:48.914191 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.914199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.915107 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.915110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.916025 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.916035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.916740 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.916869 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.916894 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.916966 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.917065 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":21}
I20260811 04:36:48.917244 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.913867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:57461 (local address 127.2.238.131:38571)
0811 04:36:48.914007 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.914012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.914152 (+ 140us) server_negotiation.cc:407] Connection header received
0811 04:36:48.914234 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.914240 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.914316 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.914382 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:48.914660 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.915192 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916255 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.916674 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916798 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.916853 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.917052 (+ 199us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.917126 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:36:48.917275 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.913629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:38571 (local address 127.2.238.130:57461)
0811 04:36:48.914053 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.914072 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:48.914156 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:48.914415 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:48.914417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:48.914435 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:48.914548 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.914554 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.915306 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.915308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.916126 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:48.916139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.916807 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.916813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:48.916908 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.916926 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:36:48.917022 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 04:36:48.917091 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"mutex_wait_us":17,"thread_start_us":185,"threads_started":1}
I20260811 04:36:48.918071 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e42055aaf2844fbb022ceeb7c58871a" candidate_uuid: "cb9caa4f155445528f2562bddbe83f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d3139af5344a29acd379a0996ee754" is_pre_election: true
I20260811 04:36:48.918066 7946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e42055aaf2844fbb022ceeb7c58871a" candidate_uuid: "cb9caa4f155445528f2562bddbe83f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e210373bd12a4b32b533e6e8b120c92e" is_pre_election: true
I20260811 04:36:48.918421 7661 raft_consensus.cc:2454] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9caa4f155445528f2562bddbe83f98 in term 0.
I20260811 04:36:48.918422 7946 raft_consensus.cc:2454] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb9caa4f155445528f2562bddbe83f98 in term 0.
I20260811 04:36:48.918970 7734 leader_election.cc:302] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9caa4f155445528f2562bddbe83f98, e210373bd12a4b32b533e6e8b120c92e; no voters:
I20260811 04:36:48.919286 8020 raft_consensus.cc:2781] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:48.919390 8020 raft_consensus.cc:482] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:48.919449 8020 raft_consensus.cc:3037] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:48.920761 8020 raft_consensus.cc:504] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.921128 8020 leader_election.cc:288] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [CANDIDATE]: Term 1 election: Requested vote from peers c0d3139af5344a29acd379a0996ee754 (127.2.238.129:40239), e210373bd12a4b32b533e6e8b120c92e (127.2.238.131:38571)
I20260811 04:36:48.921590 7661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e42055aaf2844fbb022ceeb7c58871a" candidate_uuid: "cb9caa4f155445528f2562bddbe83f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d3139af5344a29acd379a0996ee754"
I20260811 04:36:48.921591 7946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e42055aaf2844fbb022ceeb7c58871a" candidate_uuid: "cb9caa4f155445528f2562bddbe83f98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e210373bd12a4b32b533e6e8b120c92e"
I20260811 04:36:48.921754 7946 raft_consensus.cc:3037] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:48.921746 7661 raft_consensus.cc:3037] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:48.922925 7661 raft_consensus.cc:2454] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9caa4f155445528f2562bddbe83f98 in term 1.
I20260811 04:36:48.922938 7946 raft_consensus.cc:2454] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb9caa4f155445528f2562bddbe83f98 in term 1.
I20260811 04:36:48.923307 7734 leader_election.cc:302] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cb9caa4f155445528f2562bddbe83f98, e210373bd12a4b32b533e6e8b120c92e; no voters:
I20260811 04:36:48.923517 8020 raft_consensus.cc:2781] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:48.923775 8020 raft_consensus.cc:686] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 1 LEADER]: Becoming Leader. State: Replica: cb9caa4f155445528f2562bddbe83f98, State: Running, Role: LEADER
I20260811 04:36:48.924167 8020 consensus_queue.cc:237] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } }
I20260811 04:36:48.927558 7506 catalog_manager.cc:5495] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 reported cstate change: term changed from 0 to 1, leader changed from <none> to cb9caa4f155445528f2562bddbe83f98 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "cb9caa4f155445528f2562bddbe83f98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb9caa4f155445528f2562bddbe83f98" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:48.944314 8010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.940850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.2.238.130:43593)
0811 04:36:48.940986 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:48.940990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:48.941005 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:48.941141 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:48.941143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:48.941203 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:48.941294 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:48.941507 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.942074 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.942875 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:48.943077 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:48.943151 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:48.943432 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:48.943866 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:48.943958 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:48.944057 (+ 99us) server_negotiation.cc:299] Negotiation successful
0811 04:36:48.944106 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:36:48.951016 7661 raft_consensus.cc:1261] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754 [term 1 FOLLOWER]: Refusing update from remote peer cb9caa4f155445528f2562bddbe83f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:48.951016 7946 raft_consensus.cc:1261] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 1 FOLLOWER]: Refusing update from remote peer cb9caa4f155445528f2562bddbe83f98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:48.951803 8020 consensus_queue.cc:1035] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e210373bd12a4b32b533e6e8b120c92e" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 38571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:48.952023 8022 consensus_queue.cc:1035] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d3139af5344a29acd379a0996ee754" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:48.960901 8028 mvcc.cc:205] Tried to move back new op lower bound from 7317188644655308800 to 7317188644556177408. Current Snapshot: MvccSnapshot[applied={T|T < 7317188644655308800}]
I20260811 04:36:48.970340 7502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:48.970558 7502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:48.973419 7502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c6bd35a367cb46319cf22bca02bcf467: Sending DeleteTablet for 3 replicas of tablet 5e42055aaf2844fbb022ceeb7c58871a
I20260811 04:36:48.974462 7783 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:48 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:50880
I20260811 04:36:48.974529 7639 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:48 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:58648
I20260811 04:36:48.975126 8037 tablet_replica.cc:330] T 5e42055aaf2844fbb022ceeb7c58871a P c0d3139af5344a29acd379a0996ee754: stopping tablet replica
I20260811 04:36:48.975328 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7572
I20260811 04:36:48.975486 7925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e42055aaf2844fbb022ceeb7c58871a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:48 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:36202
I20260811 04:36:48.979346 8038 tablet_replica.cc:330] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: stopping tablet replica
I20260811 04:36:48.979771 8038 raft_consensus.cc:2229] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:48.980194 8038 raft_consensus.cc:2258] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:48.982059 8036 tablet_replica.cc:330] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: stopping tablet replica
I20260811 04:36:48.982079 8038 ts_tablet_manager.cc:1905] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:48.982568 8036 raft_consensus.cc:2229] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:48.983124 8036 raft_consensus.cc:2258] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:48.984961 8036 ts_tablet_manager.cc:1905] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:48.985062 8038 ts_tablet_manager.cc:1918] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:48.985170 8038 log.cc:1196] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/5e42055aaf2844fbb022ceeb7c58871a
I20260811 04:36:48.985538 8038 ts_tablet_manager.cc:1939] T 5e42055aaf2844fbb022ceeb7c58871a P e210373bd12a4b32b533e6e8b120c92e: Deleting consensus metadata
I20260811 04:36:48.986977 7490 catalog_manager.cc:4841] TS e210373bd12a4b32b533e6e8b120c92e (127.2.238.131:38571): tablet 5e42055aaf2844fbb022ceeb7c58871a (table test-table [id=db8a21ec28554a8fa5751d6d6ca6a0f4]) successfully deleted
W20260811 04:36:48.987763 7490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:40239 (error 108)
I20260811 04:36:48.988626 8036 ts_tablet_manager.cc:1918] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:48.988732 8036 log.cc:1196] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/5e42055aaf2844fbb022ceeb7c58871a
I20260811 04:36:48.988912 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:48.988595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40239 (local address 127.0.0.1:58652)
0811 04:36:48.988744 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:48.988819 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 04:36:48.989056 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7715
I20260811 04:36:48.989081 8036 ts_tablet_manager.cc:1939] T 5e42055aaf2844fbb022ceeb7c58871a P cb9caa4f155445528f2562bddbe83f98: Deleting consensus metadata
W20260811 04:36:48.992197 7490 catalog_manager.cc:4584] TS c0d3139af5344a29acd379a0996ee754 (127.2.238.129:40239): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e42055aaf2844fbb022ceeb7c58871a: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40239: connect: Connection refused (error 111)
I20260811 04:36:48.999903 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7858
I20260811 04:36:49.002617 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.000113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43593 (local address 127.0.0.1:50896)
0811 04:36:49.002430 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.002503 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.130:43593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2188}
I20260811 04:36:49.010109 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 7473
2026-08-11T04:36:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2362 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-11T04:36:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:49Z Disabled control of system clock
I20260811 04:36:49.045557 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34703
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 04:36:49.163098 8043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:49.163401 8043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:49.163441 8043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:49.167459 8043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:49.167520 8043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:49.167536 8043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:49.167553 8043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:49.167568 8043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:49.172401 8043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34703
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8043
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:49.173563 8043 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:49.175004 8043 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:49.183987 8052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.183987 8049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.183984 8050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.184433 8043 server_base.cc:1034] running on GCE node
I20260811 04:36:49.185096 8043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:49.186398 8043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:49.187589 8043 hybrid_clock.cc:648] HybridClock initialized: now 1786423009187562 us; error 49 us; skew 500 ppm
I20260811 04:36:49.189524 8043 webserver.cc:460] Webserver started at http://127.2.238.190:41323/ using document root <none> and password file <none>
I20260811 04:36:49.190284 8043 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:49.190388 8043 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:49.190652 8043 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:49.192902 8043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "fb9320e1a2e88c21656e0b5a50ac8d3c"
server_key_iv: "43c072ac1449425f54507b34a981b341"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.193526 8043 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "fb9320e1a2e88c21656e0b5a50ac8d3c"
server_key_iv: "43c072ac1449425f54507b34a981b341"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.197625 8043 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 04:36:49.200443 8059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.201910 8043 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:49.202067 8043 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "fb9320e1a2e88c21656e0b5a50ac8d3c"
server_key_iv: "43c072ac1449425f54507b34a981b341"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.202207 8043 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:49.207368 8043 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:49.208084 8043 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:49.208292 8043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:49.215860 8043 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34703
I20260811 04:36:49.215871 8112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34703 every 8 connection(s)
I20260811 04:36:49.216970 8043 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:49.220671 8113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:49.223088 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8043
I20260811 04:36:49.223248 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:49.223585 3002 external_mini_cluster.cc:1428] Setting key d1b90acb88c2a60b4f4421707a86a716
I20260811 04:36:49.227511 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: Bootstrap starting.
I20260811 04:36:49.230180 8113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:49.230188 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.225269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.2.238.190:34703)
0811 04:36:49.226116 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.226137 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.226181 (+ 44us) server_negotiation.cc:407] Connection header received
0811 04:36:49.226880 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.226898 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.227290 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.227678 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.228543 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.228549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.228562 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.228575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.228638 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.228658 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.228828 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.228925 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.229353 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":162,"threads_started":1}
I20260811 04:36:49.231240 8113 log.cc:824] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:49.233399 8113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: No bootstrap required, opened a new log
I20260811 04:36:49.237349 8113 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } }
I20260811 04:36:49.237627 8113 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:49.237675 8113 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2395a45bf1a4f2cbd4e20fdda025bfc, State: Initialized, Role: FOLLOWER
I20260811 04:36:49.238198 8113 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } }
I20260811 04:36:49.238344 8113 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:49.238392 8113 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:49.238474 8113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:49.239480 8113 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } }
I20260811 04:36:49.239831 8113 leader_election.cc:302] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2395a45bf1a4f2cbd4e20fdda025bfc; no voters:
I20260811 04:36:49.240142 8113 leader_election.cc:288] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:49.240317 8121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:49.240581 8121 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [term 1 LEADER]: Becoming Leader. State: Replica: d2395a45bf1a4f2cbd4e20fdda025bfc, State: Running, Role: LEADER
I20260811 04:36:49.240922 8121 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } }
I20260811 04:36:49.241648 8113 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:49.242830 8123 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2395a45bf1a4f2cbd4e20fdda025bfc. Latest consensus state: current_term: 1 leader_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } } }
I20260811 04:36:49.242835 8122 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2395a45bf1a4f2cbd4e20fdda025bfc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34703 } } }
I20260811 04:36:49.242990 8123 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:49.242991 8122 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:49.243395 8130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:49.246596 8130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:49.252409 8130 catalog_manager.cc:1347] Generated new cluster ID: b458e7fc9dcc4e3d95c5001e86e22028
I20260811 04:36:49.252498 8130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:49.260586 8130 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:49.260958 8130 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:49.266009 8130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: Generated new TSK 0
I20260811 04:36:49.266747 8130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:36:49.274502 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--builtin_ntp_servers=127.2.238.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260811 04:36:49.396695 8140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:49.397024 8140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:49.397068 8140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:49.401541 8140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:49.401654 8140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:49.401732 8140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:49.407158 8140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:49.408478 8140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:49.410068 8140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:49.420655 8145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.420799 8148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.420899 8140 server_base.cc:1034] running on GCE node
W20260811 04:36:49.420661 8146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.421526 8140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:49.422391 8140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:49.423593 8140 hybrid_clock.cc:648] HybridClock initialized: now 1786423009423576 us; error 29 us; skew 500 ppm
I20260811 04:36:49.425560 8140 webserver.cc:460] Webserver started at http://127.2.238.129:40359/ using document root <none> and password file <none>
I20260811 04:36:49.426265 8140 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:49.426373 8140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:49.426648 8140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:49.428918 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "198e3af76b6247fc8051e8195b218bca"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "9dc15b7b8c66ab65ec240d3a213a4ba2"
server_key_iv: "d50bd737a95ea50cafd06334fa8f47a2"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.429557 8140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "198e3af76b6247fc8051e8195b218bca"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "9dc15b7b8c66ab65ec240d3a213a4ba2"
server_key_iv: "d50bd737a95ea50cafd06334fa8f47a2"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.433879 8140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 04:36:49.436455 8154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.437709 8140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:36:49.437896 8140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "198e3af76b6247fc8051e8195b218bca"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "9dc15b7b8c66ab65ec240d3a213a4ba2"
server_key_iv: "d50bd737a95ea50cafd06334fa8f47a2"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.438023 8140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:49.443192 8140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:49.443849 8140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:49.444067 8140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:49.444676 8140 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:49.445684 8140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:49.445758 8140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.445868 8140 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:49.445912 8140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.455848 8140 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:34801
I20260811 04:36:49.455870 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:34801 every 8 connection(s)
I20260811 04:36:49.456882 8140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:49.462929 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8140
I20260811 04:36:49.463074 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:49.463476 3002 external_mini_cluster.cc:1428] Setting key b7eb7151a64c814fc60e27100b106188
I20260811 04:36:49.463879 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.458951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:39815 (local address 127.2.238.190:34703)
0811 04:36:49.459098 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.459101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.460401 (+ 1300us) server_negotiation.cc:407] Connection header received
0811 04:36:49.461967 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.461970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.462013 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.462095 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.463357 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.463360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.463362 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.463376 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.463397 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.463405 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.463491 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.463718 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.463769 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:36:49.464937 8278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.459340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34703 (local address 127.2.238.129:39815)
0811 04:36:49.460180 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.460241 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.461664 (+ 1423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.462287 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.462294 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.462389 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.463007 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.463009 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.463075 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.463175 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.463498 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.463510 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.463518 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.463749 (+ 231us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.464070 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":132,"threads_started":1}
I20260811 04:36:49.466045 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--builtin_ntp_servers=127.2.238.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 04:36:49.466380 8276 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34703
I20260811 04:36:49.466642 8276 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:49.467089 8276 heartbeater.cc:511] Master 127.2.238.190:34703 requested a full tablet report, sending...
I20260811 04:36:49.468909 8076 ts_manager.cc:194] Registered new tserver with Master: 198e3af76b6247fc8051e8195b218bca (127.2.238.129:34801)
W20260811 04:36:49.588670 8279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:49.589061 8279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:49.589165 8279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:49.593427 8279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:49.593557 8279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:49.593667 8279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:49.598877 8279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:49.600174 8279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:49.601681 8279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:49.612211 8284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.612205 8288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.612505 8279 server_base.cc:1034] running on GCE node
W20260811 04:36:49.612581 8286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.613058 8279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:49.613832 8279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:49.615046 8279 hybrid_clock.cc:648] HybridClock initialized: now 1786423009615002 us; error 55 us; skew 500 ppm
I20260811 04:36:49.617182 8279 webserver.cc:460] Webserver started at http://127.2.238.130:44655/ using document root <none> and password file <none>
I20260811 04:36:49.617915 8279 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:49.618022 8279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:49.618274 8279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:49.620571 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "a4db6b0887d64273a69fea71ab7412a4"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "e165ed41c83aa47f09e0d11aa097d8f8"
server_key_iv: "b22e5e862a45932525901bf3cf79cf2e"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.621235 8279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "a4db6b0887d64273a69fea71ab7412a4"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "e165ed41c83aa47f09e0d11aa097d8f8"
server_key_iv: "b22e5e862a45932525901bf3cf79cf2e"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.625609 8279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 04:36:49.628556 8295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.630016 8279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:49.630165 8279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "a4db6b0887d64273a69fea71ab7412a4"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "e165ed41c83aa47f09e0d11aa097d8f8"
server_key_iv: "b22e5e862a45932525901bf3cf79cf2e"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.630318 8279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:49.635448 8279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:49.636154 8279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:49.636363 8279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:49.636973 8279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:49.638038 8279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:49.638110 8279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.638195 8279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:49.638247 8279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.647925 8279 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43537
I20260811 04:36:49.647949 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43537 every 8 connection(s)
I20260811 04:36:49.649075 8279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:49.655107 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8279
I20260811 04:36:49.655268 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:49.655615 3002 external_mini_cluster.cc:1428] Setting key cb4fc76be2108e5523cafb308abdf2d2
I20260811 04:36:49.655614 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.651052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:57205 (local address 127.2.238.190:34703)
0811 04:36:49.651247 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.651252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.652365 (+ 1113us) server_negotiation.cc:407] Connection header received
0811 04:36:49.653670 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.653672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.653714 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.653810 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.655044 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.655048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.655050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.655063 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.655083 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.655091 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.655172 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.655402 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.655461 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 04:36:49.657198 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.651508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34703 (local address 127.2.238.130:57205)
0811 04:36:49.652201 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.652246 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.653469 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.654017 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.654024 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.654133 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.654690 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.654691 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.654737 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.654867 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.655186 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.655199 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.655206 (+ 7us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.655930 (+ 724us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.656186 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260811 04:36:49.658133 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--builtin_ntp_servers=127.2.238.148:42431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260811 04:36:49.658592 8414 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34703
I20260811 04:36:49.658874 8414 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:49.659327 8414 heartbeater.cc:511] Master 127.2.238.190:34703 requested a full tablet report, sending...
I20260811 04:36:49.660563 8076 ts_manager.cc:194] Registered new tserver with Master: a4db6b0887d64273a69fea71ab7412a4 (127.2.238.130:43537)
W20260811 04:36:49.779210 8417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:49.779531 8417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:49.779599 8417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:49.784184 8417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:49.784288 8417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:49.784390 8417 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:49.789444 8417 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:49.790746 8417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:49.792289 8417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:49.802611 8422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.802598 8425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:49.802598 8423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:49.802790 8417 server_base.cc:1034] running on GCE node
I20260811 04:36:49.803292 8417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:49.804008 8417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:49.805226 8417 hybrid_clock.cc:648] HybridClock initialized: now 1786423009805209 us; error 29 us; skew 500 ppm
I20260811 04:36:49.807376 8417 webserver.cc:460] Webserver started at http://127.2.238.131:34221/ using document root <none> and password file <none>
I20260811 04:36:49.808009 8417 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:49.808104 8417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:49.808393 8417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:49.810683 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "9803fa10b8fd420ca4a2364c2825adf2"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "43428d56acf947ceae8edc11d8126c88"
server_key_iv: "08f557778d1c94a21d85e89df83beec6"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.811300 8417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "9803fa10b8fd420ca4a2364c2825adf2"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "43428d56acf947ceae8edc11d8126c88"
server_key_iv: "08f557778d1c94a21d85e89df83beec6"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.815366 8417 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:49.818171 8431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.819449 8417 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:49.819557 8417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "9803fa10b8fd420ca4a2364c2825adf2"
format_stamp: "Formatted at 2026-08-11 04:36:49 on dist-test-slave-0kls"
server_key: "43428d56acf947ceae8edc11d8126c88"
server_key_iv: "08f557778d1c94a21d85e89df83beec6"
server_key_version: "encryptionkey@0"
I20260811 04:36:49.819710 8417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:49.824620 8417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:49.825260 8417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:49.825451 8417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:49.826112 8417 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:49.827090 8417 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:49.827136 8417 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.827205 8417 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:49.827244 8417 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:49.836351 8417 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:41835
I20260811 04:36:49.836370 8553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:41835 every 8 connection(s)
I20260811 04:36:49.837285 8417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:49.843799 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.839207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:49305 (local address 127.2.238.190:34703)
0811 04:36:49.839411 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.839417 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.840510 (+ 1093us) server_negotiation.cc:407] Connection header received
0811 04:36:49.841972 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.841975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.842041 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.842127 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.843382 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.843384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.843386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.843397 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.843416 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.843424 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.843501 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.843625 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.843665 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260811 04:36:49.844718 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.839672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34703 (local address 127.2.238.131:49305)
0811 04:36:49.840355 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.840397 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.841677 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.842330 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.842337 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.842428 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.843117 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.843120 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.843171 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.843245 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.843503 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.843509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.843514 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.843649 (+ 135us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.843882 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":128,"threads_started":1}
I20260811 04:36:49.845871 8554 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34703
I20260811 04:36:49.846119 8554 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:49.846659 8554 heartbeater.cc:511] Master 127.2.238.190:34703 requested a full tablet report, sending...
I20260811 04:36:49.846946 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8417
I20260811 04:36:49.847092 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:49.847397 3002 external_mini_cluster.cc:1428] Setting key 6968a77c86d36de484a4f63bf23846a2
I20260811 04:36:49.847922 8076 ts_manager.cc:194] Registered new tserver with Master: 9803fa10b8fd420ca4a2364c2825adf2 (127.2.238.131:41835)
I20260811 04:36:49.849712 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:36:49.853406 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.851740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36804 (local address 127.2.238.190:34703)
0811 04:36:49.851878 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.851894 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.852137 (+ 243us) server_negotiation.cc:407] Connection header received
0811 04:36:49.852242 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.852244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.852276 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.852343 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.853070 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.853073 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.853075 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.853086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.853106 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.853113 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.853198 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.853270 (+ 72us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.853304 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:36:49.856281 8076 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:49.859457 8076 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:49.872103 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.869196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34801 (local address 127.0.0.1:46850)
0811 04:36:49.869754 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.869767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.869867 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.870344 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.870350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.870431 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.871194 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.871196 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.871208 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.871267 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.871705 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.871716 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.871722 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.871847 (+ 125us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.871922 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":97,"threads_started":1}
I20260811 04:36:49.872370 8565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.868331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43537 (local address 127.0.0.1:55616)
0811 04:36:49.868946 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.869001 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.869125 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.870536 (+ 1411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.870539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.870558 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.871163 (+ 605us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.871165 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.871188 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.871260 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.872150 (+ 890us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.872153 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.872157 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.872221 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.872259 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":99,"threads_started":1}
I20260811 04:36:49.872659 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.869500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.2.238.129:34801)
0811 04:36:49.869936 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.869948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.869981 (+ 33us) server_negotiation.cc:407] Connection header received
0811 04:36:49.870029 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.870033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.870194 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.870305 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.871434 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.871440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.871454 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.871467 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.871515 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.871551 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.871661 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.872376 (+ 715us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.872465 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":132,"threads_started":1}
I20260811 04:36:49.873157 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.868578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41835 (local address 127.0.0.1:32922)
0811 04:36:49.869482 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.869495 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.869591 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.871562 (+ 1971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.871565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.871580 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.872213 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.872215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.872225 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.872249 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.872944 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.872948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.872951 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.873008 (+ 57us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.873041 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":94,"threads_started":1}
I20260811 04:36:49.873350 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.868464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.2.238.130:43537)
0811 04:36:49.870104 (+ 1640us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.870113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.870133 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:36:49.870180 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.870184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.870386 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.870523 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.871684 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.871690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.871700 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.871712 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.871998 (+ 286us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.872035 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.873046 (+ 1011us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.873089 (+ 43us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.873153 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":184,"threads_started":1}
I20260811 04:36:49.873646 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.869180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32922 (local address 127.2.238.131:41835)
0811 04:36:49.871171 (+ 1991us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.871185 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.871208 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:36:49.871260 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.871265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.871441 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.871581 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.872716 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.872721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.872732 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.872747 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.872794 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.872831 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.873345 (+ 514us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.873393 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.873458 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1862,"thread_start_us":116,"threads_started":1}
I20260811 04:36:49.876061 8205 tablet_service.cc:1467] Processing CreateTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e (DEFAULT_TABLE table=test-table [id=2087fb077978434b9a4bf2f8cd32ae5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:49.876207 8481 tablet_service.cc:1467] Processing CreateTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e (DEFAULT_TABLE table=test-table [id=2087fb077978434b9a4bf2f8cd32ae5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:49.876207 8330 tablet_service.cc:1467] Processing CreateTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e (DEFAULT_TABLE table=test-table [id=2087fb077978434b9a4bf2f8cd32ae5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:49.877293 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4514bcc884054088b8ab0ac6ffd24c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:49.877302 8481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4514bcc884054088b8ab0ac6ffd24c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:49.877302 8330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4514bcc884054088b8ab0ac6ffd24c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:49.883944 8571 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: Bootstrap starting.
I20260811 04:36:49.885454 8572 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Bootstrap starting.
I20260811 04:36:49.885589 8573 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Bootstrap starting.
I20260811 04:36:49.887564 8572 tablet_bootstrap.cc:654] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:49.887609 8571 tablet_bootstrap.cc:654] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:49.887609 8573 tablet_bootstrap.cc:654] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:49.888411 8572 log.cc:824] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:49.888479 8573 log.cc:824] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:49.888479 8571 log.cc:824] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:49.890739 8572 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: No bootstrap required, opened a new log
I20260811 04:36:49.890746 8571 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: No bootstrap required, opened a new log
I20260811 04:36:49.890949 8571 ts_tablet_manager.cc:1397] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 04:36:49.890949 8572 ts_tablet_manager.cc:1397] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:36:49.890739 8573 tablet_bootstrap.cc:492] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: No bootstrap required, opened a new log
I20260811 04:36:49.891337 8573 ts_tablet_manager.cc:1397] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:36:49.894712 8571 raft_consensus.cc:348] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.895007 8571 raft_consensus.cc:374] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:49.895092 8571 raft_consensus.cc:729] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198e3af76b6247fc8051e8195b218bca, State: Initialized, Role: FOLLOWER
I20260811 04:36:49.895594 8571 consensus_queue.cc:260] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.895802 8573 raft_consensus.cc:348] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.895802 8572 raft_consensus.cc:348] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.896088 8573 raft_consensus.cc:374] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:49.896087 8572 raft_consensus.cc:374] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:49.896149 8572 raft_consensus.cc:729] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4db6b0887d64273a69fea71ab7412a4, State: Initialized, Role: FOLLOWER
I20260811 04:36:49.896149 8573 raft_consensus.cc:729] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9803fa10b8fd420ca4a2364c2825adf2, State: Initialized, Role: FOLLOWER
I20260811 04:36:49.896636 8276 heartbeater.cc:503] Master 127.2.238.190:34703 was elected leader, sending a full tablet report...
I20260811 04:36:49.896590 8572 consensus_queue.cc:260] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.896589 8573 consensus_queue.cc:260] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.896802 8571 ts_tablet_manager.cc:1428] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:36:49.897560 8414 heartbeater.cc:503] Master 127.2.238.190:34703 was elected leader, sending a full tablet report...
I20260811 04:36:49.897578 8573 ts_tablet_manager.cc:1428] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:36:49.897591 8572 ts_tablet_manager.cc:1428] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 04:36:49.898124 8554 heartbeater.cc:503] Master 127.2.238.190:34703 was elected leader, sending a full tablet report...
I20260811 04:36:49.898988 8579 raft_consensus.cc:482] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:49.899472 8579 raft_consensus.cc:504] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
W20260811 04:36:49.900110 8415 tablet.cc:2369] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:36:49.901939 8579 leader_election.cc:288] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9803fa10b8fd420ca4a2364c2825adf2 (127.2.238.131:41835), 198e3af76b6247fc8051e8195b218bca (127.2.238.129:34801)
I20260811 04:36:49.902892 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.901058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34801 (local address 127.2.238.130:45393)
0811 04:36:49.901266 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.901282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.901351 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.901697 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.901700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.901716 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.902237 (+ 521us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.902239 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.902251 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.902317 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.902635 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.902638 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.902642 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.902722 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.902762 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260811 04:36:49.903349 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.901304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45393 (local address 127.2.238.129:34801)
0811 04:36:49.901407 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.901411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.901426 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:49.901466 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.901469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.901500 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.901584 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.902443 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.902447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.902449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.902480 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.902501 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.902511 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.902591 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.903156 (+ 565us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.903199 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 04:36:49.903939 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.901247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:54785 (local address 127.2.238.131:41835)
0811 04:36:49.902189 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.902193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.902582 (+ 389us) server_negotiation.cc:407] Connection header received
0811 04:36:49.902764 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.902767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.902799 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.902893 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.903529 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.903534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.903539 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.903552 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.903581 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.903591 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.903651 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.903809 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.903843 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865}
I20260811 04:36:49.903885 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4514bcc884054088b8ab0ac6ffd24c9e" candidate_uuid: "a4db6b0887d64273a69fea71ab7412a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198e3af76b6247fc8051e8195b218bca" is_pre_election: true
I20260811 04:36:49.903995 8580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.901074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:41835 (local address 127.2.238.130:54785)
0811 04:36:49.902493 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.902507 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:49.902638 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:49.902908 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:49.902911 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:49.902925 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:49.903341 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:36:49.903343 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:49.903353 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:36:49.903385 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:49.903726 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:49.903728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:49.903731 (+ 3us) client_negotiation.cc:769] Sending connection context
0811 04:36:49.903802 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 04:36:49.903835 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"mutex_wait_us":343,"thread_start_us":117,"threads_started":1}
I20260811 04:36:49.904204 8226 raft_consensus.cc:2454] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4db6b0887d64273a69fea71ab7412a4 in term 0.
I20260811 04:36:49.904567 8504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4514bcc884054088b8ab0ac6ffd24c9e" candidate_uuid: "a4db6b0887d64273a69fea71ab7412a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803fa10b8fd420ca4a2364c2825adf2" is_pre_election: true
I20260811 04:36:49.904759 8299 leader_election.cc:302] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198e3af76b6247fc8051e8195b218bca, a4db6b0887d64273a69fea71ab7412a4; no voters:
I20260811 04:36:49.904848 8504 raft_consensus.cc:2454] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4db6b0887d64273a69fea71ab7412a4 in term 0.
I20260811 04:36:49.905026 8579 raft_consensus.cc:2781] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:49.905184 8579 raft_consensus.cc:482] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:49.905252 8579 raft_consensus.cc:3037] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:49.906527 8579 raft_consensus.cc:504] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.906924 8579 leader_election.cc:288] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [CANDIDATE]: Term 1 election: Requested vote from peers 9803fa10b8fd420ca4a2364c2825adf2 (127.2.238.131:41835), 198e3af76b6247fc8051e8195b218bca (127.2.238.129:34801)
I20260811 04:36:49.907222 8504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4514bcc884054088b8ab0ac6ffd24c9e" candidate_uuid: "a4db6b0887d64273a69fea71ab7412a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9803fa10b8fd420ca4a2364c2825adf2"
I20260811 04:36:49.907292 8226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4514bcc884054088b8ab0ac6ffd24c9e" candidate_uuid: "a4db6b0887d64273a69fea71ab7412a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "198e3af76b6247fc8051e8195b218bca"
I20260811 04:36:49.907361 8504 raft_consensus.cc:3037] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:49.907418 8226 raft_consensus.cc:3037] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:49.908766 8226 raft_consensus.cc:2454] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4db6b0887d64273a69fea71ab7412a4 in term 1.
I20260811 04:36:49.908807 8504 raft_consensus.cc:2454] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4db6b0887d64273a69fea71ab7412a4 in term 1.
I20260811 04:36:49.909128 8299 leader_election.cc:302] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 198e3af76b6247fc8051e8195b218bca, a4db6b0887d64273a69fea71ab7412a4; no voters:
I20260811 04:36:49.909337 8579 raft_consensus.cc:2781] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:49.909610 8579 raft_consensus.cc:686] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 1 LEADER]: Becoming Leader. State: Replica: a4db6b0887d64273a69fea71ab7412a4, State: Running, Role: LEADER
I20260811 04:36:49.909999 8579 consensus_queue.cc:237] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } }
I20260811 04:36:49.913174 8074 catalog_manager.cc:5495] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4db6b0887d64273a69fea71ab7412a4 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "a4db6b0887d64273a69fea71ab7412a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4db6b0887d64273a69fea71ab7412a4" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:49.923974 8566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.922436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55630 (local address 127.2.238.130:43537)
0811 04:36:49.922579 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:49.922582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:49.922595 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:49.922676 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:49.922679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:49.922749 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:49.922820 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:49.923526 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:49.923529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:49.923531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:49.923542 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:49.923562 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:49.923570 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:49.923682 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:49.923758 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 04:36:49.923794 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:36:49.930430 8504 raft_consensus.cc:1261] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 1 FOLLOWER]: Refusing update from remote peer a4db6b0887d64273a69fea71ab7412a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:49.930430 8226 raft_consensus.cc:1261] T 4514bcc884054088b8ab0ac6ffd24c9e P 198e3af76b6247fc8051e8195b218bca [term 1 FOLLOWER]: Refusing update from remote peer a4db6b0887d64273a69fea71ab7412a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:49.931169 8579 consensus_queue.cc:1035] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9803fa10b8fd420ca4a2364c2825adf2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:49.931399 8581 consensus_queue.cc:1035] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "198e3af76b6247fc8051e8195b218bca" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:49.942188 8586 mvcc.cc:205] Tried to move back new op lower bound from 7317188648666927104 to 7317188648593993728. Current Snapshot: MvccSnapshot[applied={T|T < 7317188648666927104}]
I20260811 04:36:49.944947 8585 mvcc.cc:205] Tried to move back new op lower bound from 7317188648666927104 to 7317188648593993728. Current Snapshot: MvccSnapshot[applied={T|T < 7317188648666927104 or (T in {7317188648666927104})}]
I20260811 04:36:49.949484 8074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:49.949642 8074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:49.952498 8074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2395a45bf1a4f2cbd4e20fdda025bfc: Sending DeleteTablet for 3 replicas of tablet 4514bcc884054088b8ab0ac6ffd24c9e
I20260811 04:36:49.953295 8330 tablet_service.cc:1514] Processing DeleteTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:49 UTC) from {username='slave'} at 127.0.0.1:55616
I20260811 04:36:49.953379 8481 tablet_service.cc:1514] Processing DeleteTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:49 UTC) from {username='slave'} at 127.0.0.1:32922
I20260811 04:36:49.953487 8205 tablet_service.cc:1514] Processing DeleteTablet for tablet 4514bcc884054088b8ab0ac6ffd24c9e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:49 UTC) from {username='slave'} at 127.0.0.1:46850
I20260811 04:36:49.954138 8594 tablet_replica.cc:330] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: stopping tablet replica
I20260811 04:36:49.954541 8594 raft_consensus.cc:2229] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:49.954679 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8140
I20260811 04:36:49.956934 8596 tablet_replica.cc:330] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: stopping tablet replica
I20260811 04:36:49.957263 8596 raft_consensus.cc:2229] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:49.957602 8596 raft_consensus.cc:2258] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:49.955000 8594 raft_consensus.cc:2258] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:49.960019 8596 ts_tablet_manager.cc:1905] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:49.960417 8594 ts_tablet_manager.cc:1905] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:49.962595 8596 ts_tablet_manager.cc:1918] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:49.962719 8596 log.cc:1196] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/4514bcc884054088b8ab0ac6ffd24c9e
I20260811 04:36:49.963014 8596 ts_tablet_manager.cc:1939] T 4514bcc884054088b8ab0ac6ffd24c9e P 9803fa10b8fd420ca4a2364c2825adf2: Deleting consensus metadata
I20260811 04:36:49.963395 8594 ts_tablet_manager.cc:1918] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:49.963502 8594 log.cc:1196] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/4514bcc884054088b8ab0ac6ffd24c9e
I20260811 04:36:49.963850 8594 ts_tablet_manager.cc:1939] T 4514bcc884054088b8ab0ac6ffd24c9e P a4db6b0887d64273a69fea71ab7412a4: Deleting consensus metadata
I20260811 04:36:49.964404 8060 catalog_manager.cc:4841] TS 9803fa10b8fd420ca4a2364c2825adf2 (127.2.238.131:41835): tablet 4514bcc884054088b8ab0ac6ffd24c9e (table test-table [id=2087fb077978434b9a4bf2f8cd32ae5f]) successfully deleted
I20260811 04:36:49.964941 8062 catalog_manager.cc:4841] TS a4db6b0887d64273a69fea71ab7412a4 (127.2.238.130:43537): tablet 4514bcc884054088b8ab0ac6ffd24c9e (table test-table [id=2087fb077978434b9a4bf2f8cd32ae5f]) successfully deleted
W20260811 04:36:49.970727 8063 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:34801 (error 108)
I20260811 04:36:49.970870 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8279
I20260811 04:36:49.973233 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:49.972789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34801 (local address 127.0.0.1:46856)
0811 04:36:49.973028 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:49.973117 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34801: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260811 04:36:49.973449 8063 catalog_manager.cc:4584] TS 198e3af76b6247fc8051e8195b218bca (127.2.238.129:34801): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4514bcc884054088b8ab0ac6ffd24c9e: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34801: connect: Connection refused (error 111)
I20260811 04:36:49.979125 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8417
I20260811 04:36:49.987921 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8043
2026-08-11T04:36:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (976 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260811 04:36:50.012444 3002 mini_oidc.cc:152] Starting JWKS server
I20260811 04:36:50.014379 3002 webserver.cc:460] Webserver started at https://127.0.0.1:38965/ using document root <none> and password file <none>
I20260811 04:36:50.014533 3002 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 04:36:50.014770 3002 webserver.cc:460] Webserver started at http://127.0.0.1:42207/ using document root <none> and password file <none>
2026-08-11T04:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:50Z Disabled control of system clock
I20260811 04:36:50.025887 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42409
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:43843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:42409
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38965/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/testchainca.cert with env {}
W20260811 04:36:50.146108 8605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:50.146395 8605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:50.146437 8605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:50.150758 8605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:50.150839 8605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:50.150856 8605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:50.150874 8605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:50.150890 8605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 04:36:50.150906 8605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 04:36:50.150920 8605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38965/jwks.json
I20260811 04:36:50.156929 8605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:43843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:42409
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38965/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:50.158123 8605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:50.159163 8605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:50.168398 8612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:50.168440 8611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:50.168442 8614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:50.168766 8605 server_base.cc:1034] running on GCE node
I20260811 04:36:50.169432 8605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:50.170749 8605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:50.171964 8605 hybrid_clock.cc:648] HybridClock initialized: now 1786423010171943 us; error 39 us; skew 500 ppm
I20260811 04:36:50.174039 8605 webserver.cc:460] Webserver started at http://127.2.238.190:35275/ using document root <none> and password file <none>
I20260811 04:36:50.174666 8605 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:50.174767 8605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:50.175032 8605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:50.177179 8605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "74019dc4dbdb4990b83f41ed1aaa0631"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "410e9a27fb280be3b90cfe7bfe099ec9"
server_key_iv: "39a536cbe14cabc73ec1d14ab582885b"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.177933 8605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "74019dc4dbdb4990b83f41ed1aaa0631"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "410e9a27fb280be3b90cfe7bfe099ec9"
server_key_iv: "39a536cbe14cabc73ec1d14ab582885b"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.182246 8605 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:36:50.184937 8620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:50.186661 8605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:36:50.186791 8605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "74019dc4dbdb4990b83f41ed1aaa0631"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "410e9a27fb280be3b90cfe7bfe099ec9"
server_key_iv: "39a536cbe14cabc73ec1d14ab582885b"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.186908 8605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:50.199772 8605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:50.200563 8605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:50.200733 8605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:50.208557 8605 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:42409
I20260811 04:36:50.208560 8673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:42409 every 8 connection(s)
I20260811 04:36:50.209708 8605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:50.212901 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8605
I20260811 04:36:50.213047 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:50.213403 3002 external_mini_cluster.cc:1428] Setting key 6b24b00dd10221c99326d451d423b4e3
I20260811 04:36:50.213670 8674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:50.221187 8674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631: Bootstrap starting.
I20260811 04:36:50.223984 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.214929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57436 (local address 127.2.238.190:42409)
0811 04:36:50.215690 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.215712 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.215764 (+ 52us) server_negotiation.cc:407] Connection header received
0811 04:36:50.216683 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.216703 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.217178 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.217553 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:50.218081 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.219094 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.220048 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.220554 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.220742 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.221744 (+ 1002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:50.221750 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:50.221767 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:50.222326 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:50.222405 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:50.222432 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:50.222550 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:50.222671 (+ 121us) server_negotiation.cc:299] Negotiation successful
0811 04:36:50.222976 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":147,"threads_started":1}
I20260811 04:36:50.224258 8674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:50.225023 8674 log.cc:824] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:50.227232 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:36:50.227368 8674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631: No bootstrap required, opened a new log
I20260811 04:36:50.231200 8674 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } }
I20260811 04:36:50.231433 8674 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:50.231468 8674 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74019dc4dbdb4990b83f41ed1aaa0631, State: Initialized, Role: FOLLOWER
I20260811 04:36:50.231853 8674 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } }
I20260811 04:36:50.231964 8674 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:50.232009 8674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:50.232080 8674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:50.232988 8674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } }
I20260811 04:36:50.233290 8674 leader_election.cc:302] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74019dc4dbdb4990b83f41ed1aaa0631; no voters:
I20260811 04:36:50.233559 8674 leader_election.cc:288] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:50.233703 8681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:50.234009 8681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [term 1 LEADER]: Becoming Leader. State: Replica: 74019dc4dbdb4990b83f41ed1aaa0631, State: Running, Role: LEADER
I20260811 04:36:50.234349 8681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } }
I20260811 04:36:50.234846 8674 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:50.236011 8682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } } }
I20260811 04:36:50.236162 8682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.236546 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74019dc4dbdb4990b83f41ed1aaa0631. Latest consensus state: current_term: 1 leader_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74019dc4dbdb4990b83f41ed1aaa0631" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42409 } } }
I20260811 04:36:50.236632 8691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:50.236658 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.240427 8691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:50.246410 8691 catalog_manager.cc:1347] Generated new cluster ID: 5f07d9fd8ea348199be082cd3725f676
I20260811 04:36:50.246500 8691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:50.260991 8691 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:50.262228 8691 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:50.270905 8691 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74019dc4dbdb4990b83f41ed1aaa0631: Generated new TSK 0
I20260811 04:36:50.271739 8691 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 04:36:50.308184 8707 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260811 04:36:50.308393 8707 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 04:36:50.309041 8678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.289754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57438 (local address 127.2.238.190:42409)
0811 04:36:50.289947 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.289952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.290119 (+ 167us) server_negotiation.cc:407] Connection header received
0811 04:36:50.290338 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.290342 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.290403 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.290525 (+ 122us) server_negotiation.cc:226] Negotiated authn=JWT
0811 04:36:50.290794 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.291686 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.292638 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.292987 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.293092 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.293481 (+ 389us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 04:36:50.308578 (+ 15097us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 04:36:50.308888 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":88}
W20260811 04:36:50.309128 8678 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57438: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260811 04:36:50.310477 8706 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:50.289629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42409 (local address 127.0.0.1:57438)
0811 04:36:50.290025 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:50.290038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:50.290162 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:50.290553 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:50.290555 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:50.290564 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 04:36:50.290682 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.290687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.291816 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.291819 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.292514 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.292524 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.293129 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.293132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.293204 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.293265 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 04:36:50.308858 (+ 15593us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0811 04:36:50.309074 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.238.190:42409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":144,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.238.190:42409: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38965/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260811 04:36:50.311347 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8605
2026-08-11T04:36:50Z chronyd exiting
I20260811 04:36:50.352922 3002 test_util.cc:175] -----------------------------------------------
I20260811 04:36:50.353034 3002 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786422982503306-3002-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (342 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260811 04:36:50.354372 3002 mini_oidc.cc:152] Starting JWKS server
I20260811 04:36:50.355319 3002 webserver.cc:460] Webserver started at https://127.0.0.1:42989/ using document root <none> and password file <none>
I20260811 04:36:50.355455 3002 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 04:36:50.355732 3002 webserver.cc:460] Webserver started at http://127.0.0.1:41153/ using document root <none> and password file <none>
2026-08-11T04:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:50Z Disabled control of system clock
I20260811 04:36:50.366801 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:37921
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:39953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:37921
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42989/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/testchainca.cert with env {}
W20260811 04:36:50.486248 8713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:50.486557 8713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:50.486598 8713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:50.490640 8713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:50.490731 8713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:50.490746 8713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:50.490772 8713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:50.490835 8713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 04:36:50.490867 8713 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 04:36:50.490899 8713 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42989/jwks.json
I20260811 04:36:50.496716 8713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:37921
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42989/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:50.498081 8713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:50.499208 8713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:50.508446 8719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:50.508486 8720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:50.508450 8722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:50.508823 8713 server_base.cc:1034] running on GCE node
I20260811 04:36:50.509493 8713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:50.510787 8713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:50.511983 8713 hybrid_clock.cc:648] HybridClock initialized: now 1786423010511970 us; error 27 us; skew 500 ppm
I20260811 04:36:50.514096 8713 webserver.cc:460] Webserver started at http://127.2.238.190:45845/ using document root <none> and password file <none>
I20260811 04:36:50.514734 8713 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:50.514837 8713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:50.515090 8713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:50.517422 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "8daf647d15614338a357d534c5037869"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "5eba3deb99acb5f9a4537e07421befbf"
server_key_iv: "7b0afa12df73a9a4936ba69d4e1a2c43"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.518177 8713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "8daf647d15614338a357d534c5037869"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "5eba3deb99acb5f9a4537e07421befbf"
server_key_iv: "7b0afa12df73a9a4936ba69d4e1a2c43"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.523581 8713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260811 04:36:50.528009 8728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:50.529906 8713 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:50.530071 8713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "8daf647d15614338a357d534c5037869"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "5eba3deb99acb5f9a4537e07421befbf"
server_key_iv: "7b0afa12df73a9a4936ba69d4e1a2c43"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.530205 8713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:50.547495 8713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:50.548276 8713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:50.548503 8713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:50.556406 8713 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:37921
I20260811 04:36:50.556422 8783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:37921 every 8 connection(s)
I20260811 04:36:50.557436 8713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:50.560796 8784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:50.563588 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8713
I20260811 04:36:50.563747 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:50.564049 3002 external_mini_cluster.cc:1428] Setting key 749017c1b3869fd38e79542d6831c595
I20260811 04:36:50.567435 8784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869: Bootstrap starting.
I20260811 04:36:50.570360 8784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:50.571256 8784 log.cc:824] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:50.573632 8784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869: No bootstrap required, opened a new log
I20260811 04:36:50.574115 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.565539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.2.238.190:37921)
0811 04:36:50.566234 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.566252 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.566296 (+ 44us) server_negotiation.cc:407] Connection header received
0811 04:36:50.566945 (+ 649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.566963 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.567394 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.567750 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:50.568132 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.569122 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.570416 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.570961 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.571100 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.572035 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:50.572045 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:50.572066 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:50.572565 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:50.572625 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:50.572648 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:50.572731 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:50.572905 (+ 174us) server_negotiation.cc:299] Negotiation successful
0811 04:36:50.573227 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":184,"threads_started":1}
I20260811 04:36:50.577385 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:36:50.578161 8784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } }
I20260811 04:36:50.578527 8784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:50.578634 8784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8daf647d15614338a357d534c5037869, State: Initialized, Role: FOLLOWER
I20260811 04:36:50.579125 8784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } }
I20260811 04:36:50.579298 8784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:50.579373 8784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:50.579491 8784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:50.580439 8784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } }
I20260811 04:36:50.580849 8784 leader_election.cc:302] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8daf647d15614338a357d534c5037869; no voters:
I20260811 04:36:50.581161 8784 leader_election.cc:288] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:50.581323 8789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:50.581580 8789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [term 1 LEADER]: Becoming Leader. State: Replica: 8daf647d15614338a357d534c5037869, State: Running, Role: LEADER
I20260811 04:36:50.581941 8789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } }
I20260811 04:36:50.582522 8784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:50.583693 8791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8daf647d15614338a357d534c5037869. Latest consensus state: current_term: 1 leader_uuid: "8daf647d15614338a357d534c5037869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } } }
I20260811 04:36:50.583847 8791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.583709 8790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8daf647d15614338a357d534c5037869" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8daf647d15614338a357d534c5037869" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 37921 } } }
I20260811 04:36:50.584007 8790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.584337 8800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:50.587783 8800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:50.594360 8800 catalog_manager.cc:1347] Generated new cluster ID: 24db214e677143cca56bc55c582c30e4
I20260811 04:36:50.594468 8800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:50.618149 8800 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:50.619217 8800 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:50.623852 8800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8daf647d15614338a357d534c5037869: Generated new TSK 0
I20260811 04:36:50.624575 8800 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260811 04:36:50.656447 8816 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260811 04:36:50.656548 8816 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260811 04:36:50.657344 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.640753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53224 (local address 127.2.238.190:37921)
0811 04:36:50.640919 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.640924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.641129 (+ 205us) server_negotiation.cc:407] Connection header received
0811 04:36:50.641311 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.641313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.641364 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.641460 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0811 04:36:50.641732 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.642353 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.643608 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.643835 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.643919 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.644273 (+ 354us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 04:36:50.656953 (+ 12680us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 04:36:50.657216 (+ 263us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 04:36:50.657440 8787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53224: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260811 04:36:50.657615 8815 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:50.640665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:37921 (local address 127.0.0.1:53224)
0811 04:36:50.641018 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:50.641034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:50.641163 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:50.641482 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:50.641485 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:50.641494 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0811 04:36:50.641622 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.641628 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.642467 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.642470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.643469 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.643481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.643958 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.643962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.644057 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.644089 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 04:36:50.657197 (+ 13108us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0811 04:36:50.657421 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.238.190:37921: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42989/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":149,"threads_started":1}
I20260811 04:36:50.658030 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8713
2026-08-11T04:36:50Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (347 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260811 04:36:50.702316 3002 mini_oidc.cc:152] Starting JWKS server
I20260811 04:36:50.703183 3002 webserver.cc:460] Webserver started at https://127.0.0.1:46141/ using document root <none> and password file <none>
I20260811 04:36:50.703267 3002 mini_oidc.cc:189] Starting OIDC Discovery server
I20260811 04:36:50.703475 3002 webserver.cc:460] Webserver started at http://127.0.0.1:44639/ using document root <none> and password file <none>
2026-08-11T04:36:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:50Z Disabled control of system clock
I20260811 04:36:50.714762 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43961
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:37847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:43961
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46141/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 04:36:50.831593 8823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:50.831867 8823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:50.831909 8823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:50.835867 8823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:50.835930 8823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:50.835945 8823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:50.835963 8823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:50.835978 8823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 04:36:50.835994 8823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 04:36:50.836007 8823 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46141/jwks.json
I20260811 04:36:50.840755 8823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:43961
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46141/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:50.841897 8823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:50.843281 8823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:50.852108 8832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:50.852128 8829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:50.852177 8823 server_base.cc:1034] running on GCE node
W20260811 04:36:50.852123 8830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:50.853039 8823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:50.854326 8823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:50.855511 8823 hybrid_clock.cc:648] HybridClock initialized: now 1786423010855499 us; error 29 us; skew 500 ppm
I20260811 04:36:50.860711 8823 webserver.cc:460] Webserver started at http://127.2.238.190:35125/ using document root <none> and password file <none>
I20260811 04:36:50.861402 8823 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:50.861471 8823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:50.861732 8823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:50.864041 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "6398a8b245114fc38d709d1ffefef73a"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "602113f2508bddc0b219d082dfeedae4"
server_key_iv: "bde8471bc1f754351002e1a4ead30cc3"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.864674 8823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "6398a8b245114fc38d709d1ffefef73a"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "602113f2508bddc0b219d082dfeedae4"
server_key_iv: "bde8471bc1f754351002e1a4ead30cc3"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.868729 8823 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:50.871430 8838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:50.873032 8823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:50.873195 8823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "6398a8b245114fc38d709d1ffefef73a"
format_stamp: "Formatted at 2026-08-11 04:36:50 on dist-test-slave-0kls"
server_key: "602113f2508bddc0b219d082dfeedae4"
server_key_iv: "bde8471bc1f754351002e1a4ead30cc3"
server_key_version: "encryptionkey@0"
I20260811 04:36:50.873317 8823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:50.893966 8823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:50.894761 8823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:50.894986 8823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:50.902209 8823 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:43961
I20260811 04:36:50.902235 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:43961 every 8 connection(s)
I20260811 04:36:50.903271 8823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:50.906845 8896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:50.912237 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8823
I20260811 04:36:50.912360 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:50.912645 3002 external_mini_cluster.cc:1428] Setting key 4a0b39d87aa1f7ea9833faa8f5c4f0ce
I20260811 04:36:50.912847 8896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a: Bootstrap starting.
I20260811 04:36:50.915940 8896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:50.916893 8896 log.cc:824] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:50.919385 8896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a: No bootstrap required, opened a new log
I20260811 04:36:50.924199 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.914231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33188 (local address 127.2.238.190:43961)
0811 04:36:50.915002 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.915022 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.915068 (+ 46us) server_negotiation.cc:407] Connection header received
0811 04:36:50.916029 (+ 961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.916055 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.916594 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.917002 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:50.917471 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.918927 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.919941 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.920661 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.920871 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.921745 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:50.921752 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:50.921771 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:50.922343 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:50.922420 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:50.922447 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:50.922551 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:50.922690 (+ 139us) server_negotiation.cc:299] Negotiation successful
0811 04:36:50.923053 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":133,"threads_started":1}
I20260811 04:36:50.924309 8896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } }
I20260811 04:36:50.924594 8896 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:50.924679 8896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6398a8b245114fc38d709d1ffefef73a, State: Initialized, Role: FOLLOWER
I20260811 04:36:50.925128 8896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } }
I20260811 04:36:50.925264 8896 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:50.925355 8896 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:50.925467 8896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:50.926810 8896 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } }
I20260811 04:36:50.927263 8896 leader_election.cc:302] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6398a8b245114fc38d709d1ffefef73a; no voters:
I20260811 04:36:50.927664 8896 leader_election.cc:288] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:50.927688 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:36:50.927780 8902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:50.927987 8902 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [term 1 LEADER]: Becoming Leader. State: Replica: 6398a8b245114fc38d709d1ffefef73a, State: Running, Role: LEADER
I20260811 04:36:50.928349 8902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } }
I20260811 04:36:50.929147 8896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:50.930398 8903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6398a8b245114fc38d709d1ffefef73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } } }
I20260811 04:36:50.930444 8904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6398a8b245114fc38d709d1ffefef73a. Latest consensus state: current_term: 1 leader_uuid: "6398a8b245114fc38d709d1ffefef73a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6398a8b245114fc38d709d1ffefef73a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43961 } } }
I20260811 04:36:50.930563 8903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.930564 8904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:50.931066 8911 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:50.934469 8911 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:50.940593 8911 catalog_manager.cc:1347] Generated new cluster ID: 6848f67019ee49c8b2cd068d7a716a98
I20260811 04:36:50.940668 8911 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:50.950157 8911 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:50.951443 8911 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:50.958736 8911 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6398a8b245114fc38d709d1ffefef73a: Generated new TSK 0
I20260811 04:36:50.959528 8911 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:36:50.978190 8900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:50.972620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.2.238.190:43961)
0811 04:36:50.972770 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:50.972786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:50.972963 (+ 177us) server_negotiation.cc:407] Connection header received
0811 04:36:50.973136 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:50.973138 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:50.973190 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:50.973286 (+ 96us) server_negotiation.cc:226] Negotiated authn=JWT
0811 04:36:50.973544 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.974182 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.975488 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.975702 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.975781 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.976197 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 04:36:50.977768 (+ 1571us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 04:36:50.978043 (+ 275us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":56}
W20260811 04:36:50.978274 8900 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33198: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260811 04:36:50.980036 8928 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:50.972538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:43961 (local address 127.0.0.1:33198)
0811 04:36:50.972864 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:50.972880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:50.973004 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:50.973306 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:50.973309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:50.973317 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0811 04:36:50.973437 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.973444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.974319 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.974323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.975360 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:50.975371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:50.975819 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:50.975823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:50.975930 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:50.975961 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 04:36:50.978453 (+ 2492us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0811 04:36:50.979852 (+ 1399us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.238.190:43961: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46141/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":1173,"thread_start_us":126,"threads_started":1}
I20260811 04:36:50.980577 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8823
E20260811 04:36:50.990636 8929 webserver.cc:533] Webserver: sslize failed:
E20260811 04:36:50.990720 8929 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-11T04:36:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (320 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-11T04:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:51Z Disabled control of system clock
I20260811 04:36:51.033702 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41589
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:34839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:41589
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/testchainca.cert with env {}
W20260811 04:36:51.153894 8933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:51.154238 8933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:51.154342 8933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:51.158450 8933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:51.158555 8933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:51.158586 8933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:51.158631 8933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:51.158684 8933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260811 04:36:51.158721 8933 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260811 04:36:51.158752 8933 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260811 04:36:51.164489 8933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:34839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:41589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:51.165777 8933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:51.166970 8933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:51.176307 8938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:51.176337 8939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:51.176357 8941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:51.176510 8933 server_base.cc:1034] running on GCE node
I20260811 04:36:51.177384 8933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:51.178655 8933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:51.179819 8933 hybrid_clock.cc:648] HybridClock initialized: now 1786423011179806 us; error 29 us; skew 500 ppm
I20260811 04:36:51.181896 8933 webserver.cc:460] Webserver started at http://127.2.238.190:33913/ using document root <none> and password file <none>
I20260811 04:36:51.182482 8933 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:51.182546 8933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:51.182727 8933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:51.184859 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c"
format_stamp: "Formatted at 2026-08-11 04:36:51 on dist-test-slave-0kls"
server_key: "703277ac99d2f06fa3329707e880e052"
server_key_iv: "88f9e586c3d98ac3f133a66daf9cabc5"
server_key_version: "encryptionkey@0"
I20260811 04:36:51.185411 8933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c"
format_stamp: "Formatted at 2026-08-11 04:36:51 on dist-test-slave-0kls"
server_key: "703277ac99d2f06fa3329707e880e052"
server_key_iv: "88f9e586c3d98ac3f133a66daf9cabc5"
server_key_version: "encryptionkey@0"
I20260811 04:36:51.189301 8933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:51.192103 8948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:51.193392 8933 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:51.193501 8933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c"
format_stamp: "Formatted at 2026-08-11 04:36:51 on dist-test-slave-0kls"
server_key: "703277ac99d2f06fa3329707e880e052"
server_key_iv: "88f9e586c3d98ac3f133a66daf9cabc5"
server_key_version: "encryptionkey@0"
I20260811 04:36:51.193599 8933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:51.224965 8933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:51.225744 8933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:51.225940 8933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:51.233500 8933 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:41589
I20260811 04:36:51.233511 9001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:41589 every 8 connection(s)
I20260811 04:36:51.234587 8933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:51.237980 9002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:51.241129 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 8933
I20260811 04:36:51.241308 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:51.241602 3002 external_mini_cluster.cc:1428] Setting key 5a185d86b3f8da458918bd2dc2aaca78
I20260811 04:36:51.244428 9002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c: Bootstrap starting.
I20260811 04:36:51.246940 9002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:51.247759 9002 log.cc:824] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:51.250094 9002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c: No bootstrap required, opened a new log
I20260811 04:36:51.251473 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:51.243122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.2.238.190:41589)
0811 04:36:51.243826 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:51.243843 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:51.243881 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:36:51.244513 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:51.244531 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:51.244997 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:51.245343 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:51.245866 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.246879 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.247839 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.248371 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.248553 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:51.249490 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:51.249495 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:51.249509 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:36:51.249992 (+ 483us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:51.250056 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:36:51.250079 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:51.250187 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:51.250363 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 04:36:51.250635 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":138,"threads_started":1}
I20260811 04:36:51.254596 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:36:51.254828 9002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } }
I20260811 04:36:51.255100 9002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:51.255193 9002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d22a4a0b4cc4f5abb90c5f9be92cb2c, State: Initialized, Role: FOLLOWER
I20260811 04:36:51.255722 9002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } }
I20260811 04:36:51.255927 9002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:51.256004 9002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:51.256111 9002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:51.257306 9002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } }
I20260811 04:36:51.257696 9002 leader_election.cc:302] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d22a4a0b4cc4f5abb90c5f9be92cb2c; no voters:
I20260811 04:36:51.258086 9002 leader_election.cc:288] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:51.258251 9007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:51.258510 9007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [term 1 LEADER]: Becoming Leader. State: Replica: 7d22a4a0b4cc4f5abb90c5f9be92cb2c, State: Running, Role: LEADER
I20260811 04:36:51.258831 9007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } }
I20260811 04:36:51.259440 9002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:51.260730 9009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d22a4a0b4cc4f5abb90c5f9be92cb2c. Latest consensus state: current_term: 1 leader_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } } }
I20260811 04:36:51.260883 9009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:51.261125 9008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d22a4a0b4cc4f5abb90c5f9be92cb2c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41589 } } }
I20260811 04:36:51.261237 9008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:51.261369 9017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:51.264640 9017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:51.270545 9017 catalog_manager.cc:1347] Generated new cluster ID: ab664e9231474becb961ac4465bf64e0
I20260811 04:36:51.270623 9017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:51.286911 9017 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:51.288192 9017 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:51.293191 9017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d22a4a0b4cc4f5abb90c5f9be92cb2c: Generated new TSK 0
I20260811 04:36:51.293994 9017 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:36:51.350423 9005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:51.304835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44646 (local address 127.2.238.190:41589)
0811 04:36:51.304969 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:51.304973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:51.305117 (+ 144us) server_negotiation.cc:407] Connection header received
0811 04:36:51.305300 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:51.305302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:51.305350 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:51.305433 (+ 83us) server_negotiation.cc:226] Negotiated authn=JWT
0811 04:36:51.305711 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.306322 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.307588 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.307803 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.307877 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:51.308248 (+ 371us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0811 04:36:51.350009 (+ 41761us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 04:36:51.350297 (+ 288us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44646: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260811 04:36:51.350535 9005 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44646: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260811 04:36:51.350695 9033 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:51.304686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41589 (local address 127.0.0.1:44646)
0811 04:36:51.305018 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:51.305034 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:51.305157 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:51.305461 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:51.305464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:51.305472 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0811 04:36:51.305601 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:51.305608 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.306444 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.306450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:51.307473 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:51.307484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:51.307922 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:51.307926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:51.308028 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:51.308072 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0811 04:36:51.350268 (+ 42196us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0811 04:36:51.350496 (+ 228us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.238.190:41589: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":105,"threads_started":1}
I20260811 04:36:51.351639 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 8933
2026-08-11T04:36:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (351 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (133 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (135 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:51 dist-test-slave-0kls krb5kdc[9046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:51 dist-test-slave-0kls krb5kdc[9046](info): set up 2 sockets
Aug 11 04:36:51 dist-test-slave-0kls krb5kdc[9046](info): commencing operation
krb5kdc: starting...
W20260811 04:36:52.703835 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:52 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:36:52.840742 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:38261
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:38261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:52.959945 9059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:52.960249 9059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:52.960331 9059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:52.964368 9059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:52.964440 9059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:52.964460 9059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:52.964478 9059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:52.964493 9059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:52.970213 9059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:38261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:38261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:52.971421 9059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:52.972523 9059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:52.981657 9064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:52.981734 9059 server_base.cc:1034] running on GCE node
W20260811 04:36:52.981658 9065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:52.981657 9067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:52.982564 9059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:52.983736 9059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:52.984908 9059 hybrid_clock.cc:648] HybridClock initialized: now 1786423012984894 us; error 30 us; skew 500 ppm
Aug 11 04:36:52 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423012, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:52.987890 9059 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:36:52.988876 9059 webserver.cc:460] Webserver started at http://127.2.238.190:39677/ using document root <none> and password file <none>
I20260811 04:36:52.989491 9059 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:52.989557 9059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:52.989877 9059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:52.992027 9059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "666c915b67b441abb227557e939246e5"
format_stamp: "Formatted at 2026-08-11 04:36:52 on dist-test-slave-0kls"
server_key: "5cd9d6842299b852a2a2755362287e83"
server_key_iv: "2ff997a88aaecc17bf746edcda169821"
server_key_version: "encryptionkey@0"
I20260811 04:36:52.992594 9059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "666c915b67b441abb227557e939246e5"
format_stamp: "Formatted at 2026-08-11 04:36:52 on dist-test-slave-0kls"
server_key: "5cd9d6842299b852a2a2755362287e83"
server_key_iv: "2ff997a88aaecc17bf746edcda169821"
server_key_version: "encryptionkey@0"
I20260811 04:36:52.996518 9059 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 04:36:52.999125 9075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.000546 9059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:53.000707 9059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "666c915b67b441abb227557e939246e5"
format_stamp: "Formatted at 2026-08-11 04:36:52 on dist-test-slave-0kls"
server_key: "5cd9d6842299b852a2a2755362287e83"
server_key_iv: "2ff997a88aaecc17bf746edcda169821"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.000829 9059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:53.023119 9059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:53.026250 9059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:53.026473 9059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:53.033874 9059 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:38261
I20260811 04:36:53.033902 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:38261 every 8 connection(s)
I20260811 04:36:53.034885 9059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:53.038007 9132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:53.038659 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9059
I20260811 04:36:53.038793 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:53.039108 3002 external_mini_cluster.cc:1428] Setting key 76f3fcae08b3927888885f79480254a9
I20260811 04:36:53.045152 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5: Bootstrap starting.
I20260811 04:36:53.048038 9132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:53.048939 9132 log.cc:824] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:53.051383 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5: No bootstrap required, opened a new log
I20260811 04:36:53.054637 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.040563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.2.238.190:38261)
0811 04:36:53.041041 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.041060 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.041187 (+ 127us) server_negotiation.cc:407] Connection header received
0811 04:36:53.042113 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.042139 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.042572 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.042965 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.043717 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.045221 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.046195 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.046909 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.047076 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.049943 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.049949 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.049965 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.049993 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.051710 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.052200 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.052204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.052210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.052341 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.052636 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.052639 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.052640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.053061 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.053212 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.053538 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.053886 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":120,"threads_started":1}
I20260811 04:36:53.055644 9132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } }
I20260811 04:36:53.055872 9132 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:53.055938 9132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666c915b67b441abb227557e939246e5, State: Initialized, Role: FOLLOWER
I20260811 04:36:53.056380 9132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } }
I20260811 04:36:53.056501 9132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:53.056602 9132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:53.056716 9132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:53.057713 9132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } }
I20260811 04:36:53.058094 9132 leader_election.cc:302] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666c915b67b441abb227557e939246e5; no voters:
I20260811 04:36:53.058434 9132 leader_election.cc:288] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:53.058566 9137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:53.058808 9137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [term 1 LEADER]: Becoming Leader. State: Replica: 666c915b67b441abb227557e939246e5, State: Running, Role: LEADER
I20260811 04:36:53.059149 9137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } }
I20260811 04:36:53.059706 9132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:53.060930 9138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666c915b67b441abb227557e939246e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } } }
I20260811 04:36:53.061095 9138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:53.061429 9139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666c915b67b441abb227557e939246e5. Latest consensus state: current_term: 1 leader_uuid: "666c915b67b441abb227557e939246e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666c915b67b441abb227557e939246e5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38261 } } }
I20260811 04:36:53.061517 9139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:53.061571 9146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:53.065330 9146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:53.071715 9146 catalog_manager.cc:1347] Generated new cluster ID: f15534ce7ce0401f8f56890c535aef2b
I20260811 04:36:53.071805 9146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:53.089236 9146 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:53.090523 9146 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:53.096391 9146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666c915b67b441abb227557e939246e5: Generated new TSK 0
I20260811 04:36:53.097178 9146 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:36:53.136590 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:53.270604 9158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:53.270897 9158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:53.270943 9158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:53.275095 9158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:53.275157 9158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:53.275257 9158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:53.280720 9158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:53.281901 9158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:53.282979 9158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:53.293177 9163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:53.293102 9166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:53.293099 9164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:53.293094 9158 server_base.cc:1034] running on GCE node
I20260811 04:36:53.293718 9158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:53.294445 9158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:53.295636 9158 hybrid_clock.cc:648] HybridClock initialized: now 1786423013295603 us; error 46 us; skew 500 ppm
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.298411 9158 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:36:53.299341 9158 webserver.cc:460] Webserver started at http://127.2.238.129:43885/ using document root <none> and password file <none>
I20260811 04:36:53.299942 9158 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:53.300038 9158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:53.300288 9158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:53.302466 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "88990cd3747f447986aa097c79a1ed0a"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "5e58b8a0158e980e25a07639b8038510"
server_key_iv: "dfba90090de97bd177ab2801adbb5d47"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.303082 9158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "88990cd3747f447986aa097c79a1ed0a"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "5e58b8a0158e980e25a07639b8038510"
server_key_iv: "dfba90090de97bd177ab2801adbb5d47"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.307116 9158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:36:53.309562 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.310905 9158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:53.311046 9158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "88990cd3747f447986aa097c79a1ed0a"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "5e58b8a0158e980e25a07639b8038510"
server_key_iv: "dfba90090de97bd177ab2801adbb5d47"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.311163 9158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:53.324605 9158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:53.327639 9158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:53.327872 9158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:53.328509 9158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:53.329419 9158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:53.329493 9158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.329569 9158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:53.329625 9158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.339458 9158 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44989
I20260811 04:36:53.339500 9297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44989 every 8 connection(s)
I20260811 04:36:53.340448 9158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:53.345571 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9158
I20260811 04:36:53.345683 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:53.346035 3002 external_mini_cluster.cc:1428] Setting key 7472928a3fa4b2240f8a5c139229af3a
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:53.355346 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.342342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:47933 (local address 127.2.238.190:38261)
0811 04:36:53.342489 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.342493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.343361 (+ 868us) server_negotiation.cc:407] Connection header received
0811 04:36:53.344866 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.344869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.344942 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.345021 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.346432 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.347176 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.348478 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.348726 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.348852 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.351692 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.351696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.351698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.351726 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.353168 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.353716 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.353720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.353722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.353854 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.354221 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.354224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.354226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.354425 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.354533 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.355045 (+ 512us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.355179 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:53.356374 9300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.342738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38261 (local address 127.2.238.129:47933)
0811 04:36:53.343198 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:53.343243 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:53.344635 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:53.345198 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:53.345205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:53.345594 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:53.346282 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.346298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.347440 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.347443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.348347 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.348357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.348899 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.348902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.349082 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.349718 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:53.349736 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:53.351521 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:53.353303 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.353309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.353326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.353597 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.353956 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.353959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.353961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.354112 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.354535 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:53.354544 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:53.354807 (+ 263us) client_negotiation.cc:769] Sending connection context
0811 04:36:53.355004 (+ 197us) client_negotiation.cc:240] Negotiation successful
0811 04:36:53.355360 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260811 04:36:53.357661 9298 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38261
I20260811 04:36:53.357955 9298 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:53.358558 9298 heartbeater.cc:511] Master 127.2.238.190:38261 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:36:53.360260 9093 ts_manager.cc:194] Registered new tserver with Master: 88990cd3747f447986aa097c79a1ed0a (127.2.238.129:44989)
I20260811 04:36:53.361482 9093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:47933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:36:53.376080 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:53.492071 9304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:53.492348 9304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:53.492432 9304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:53.496605 9304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:53.496680 9304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:53.496767 9304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:53.502676 9304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:53.503948 9304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:53.505131 9304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:53.515224 9309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:53.515519 9304 server_base.cc:1034] running on GCE node
W20260811 04:36:53.515255 9312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:53.515239 9310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:53.516134 9304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:53.516757 9304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:53.517905 9304 hybrid_clock.cc:648] HybridClock initialized: now 1786423013517888 us; error 27 us; skew 500 ppm
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.520828 9304 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:36:53.521727 9304 webserver.cc:460] Webserver started at http://127.2.238.130:43925/ using document root <none> and password file <none>
I20260811 04:36:53.522358 9304 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:53.522483 9304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:53.522723 9304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:53.524950 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "9039a4a82a66491eb8d1e6b7ac40605d"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "a9368797cd7b578308384890c0662d55"
server_key_iv: "7c3f6e86c73f6f6e69344d203b03881e"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.525542 9304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "9039a4a82a66491eb8d1e6b7ac40605d"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "a9368797cd7b578308384890c0662d55"
server_key_iv: "7c3f6e86c73f6f6e69344d203b03881e"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.529561 9304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 04:36:53.532260 9319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.533568 9304 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:53.533670 9304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "9039a4a82a66491eb8d1e6b7ac40605d"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "a9368797cd7b578308384890c0662d55"
server_key_iv: "7c3f6e86c73f6f6e69344d203b03881e"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.533758 9304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:53.547605 9304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:53.550420 9304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:53.550590 9304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:53.551147 9304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:53.551924 9304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:53.551970 9304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.552011 9304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:53.552031 9304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.560874 9304 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:37077
I20260811 04:36:53.560904 9437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:37077 every 8 connection(s)
I20260811 04:36:53.561884 9304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:53.563472 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9304
I20260811 04:36:53.563596 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:53.563906 3002 external_mini_cluster.cc:1428] Setting key 831cadbde7517da9221262baea4c077f
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:36:53.577360 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.563767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:58941 (local address 127.2.238.190:38261)
0811 04:36:53.563924 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.563929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.564949 (+ 1020us) server_negotiation.cc:407] Connection header received
0811 04:36:53.566609 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.566615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.566684 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.566787 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.567960 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.568840 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.570119 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.570559 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.570693 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.573608 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.573616 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.573619 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.573652 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.575314 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.575806 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.575808 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.575809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.575884 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.576265 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.576267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.576268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.576456 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.576532 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.577036 (+ 504us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.577141 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
I20260811 04:36:53.578459 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.564248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38261 (local address 127.2.238.130:58941)
0811 04:36:53.564786 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:53.564843 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:53.566350 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:53.566972 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:53.566979 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:53.567329 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:53.567812 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.567824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.568982 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.568985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.569989 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.569999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.570761 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.570764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.570940 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.571648 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:53.571665 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:53.573417 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:53.575432 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.575437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.575452 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.575715 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.575966 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.575968 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.575969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.576147 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.576550 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:53.576556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:53.576818 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 04:36:53.576985 (+ 167us) client_negotiation.cc:240] Negotiation successful
0811 04:36:53.577329 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260811 04:36:53.581105 9438 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38261
I20260811 04:36:53.581390 9438 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:53.582010 9438 heartbeater.cc:511] Master 127.2.238.190:38261 requested a full tablet report, sending...
I20260811 04:36:53.583128 9093 ts_manager.cc:194] Registered new tserver with Master: 9039a4a82a66491eb8d1e6b7ac40605d (127.2.238.130:37077)
I20260811 04:36:53.583814 9093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:58941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:36:53.596280 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:53.717708 9443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:53.718024 9443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:53.718070 9443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:53.722287 9443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:53.722363 9443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:53.722426 9443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:53.727973 9443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:53.729117 9443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:53.730270 9443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:53.741066 9451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:53.741065 9449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:53.741110 9448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:53.741542 9443 server_base.cc:1034] running on GCE node
I20260811 04:36:53.742069 9443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:53.742799 9443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:53.744004 9443 hybrid_clock.cc:648] HybridClock initialized: now 1786423013743971 us; error 50 us; skew 500 ppm
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.746984 9443 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:36:53.747959 9443 webserver.cc:460] Webserver started at http://127.2.238.131:41739/ using document root <none> and password file <none>
I20260811 04:36:53.748560 9443 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:53.748683 9443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:53.748940 9443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:53.751336 9443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "1951f031ee93470884ee8a5e513be253"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "1e00b109f0807f6861f3e98e768a5666"
server_key_iv: "49c6592a74fdf957be5d35f1c1c1461b"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.751999 9443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "1951f031ee93470884ee8a5e513be253"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "1e00b109f0807f6861f3e98e768a5666"
server_key_iv: "49c6592a74fdf957be5d35f1c1c1461b"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.756409 9443 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:53.759169 9458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.760627 9443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 04:36:53.760792 9443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "1951f031ee93470884ee8a5e513be253"
format_stamp: "Formatted at 2026-08-11 04:36:53 on dist-test-slave-0kls"
server_key: "1e00b109f0807f6861f3e98e768a5666"
server_key_iv: "49c6592a74fdf957be5d35f1c1c1461b"
server_key_version: "encryptionkey@0"
I20260811 04:36:53.760888 9443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:53.774027 9443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:53.777397 9443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:53.777555 9443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:53.778317 9443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:53.779322 9443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:53.779372 9443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.779440 9443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:53.779474 9443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:53.788838 9443 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:36063
I20260811 04:36:53.788856 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:36063 every 8 connection(s)
I20260811 04:36:53.790006 9443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:53.794373 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9443
I20260811 04:36:53.794515 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:53.794872 3002 external_mini_cluster.cc:1428] Setting key 342a9b23daaa55424bd9c3a45ca07c4c
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:36:53.804802 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.791963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40149 (local address 127.2.238.190:38261)
0811 04:36:53.792171 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.792175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.793047 (+ 872us) server_negotiation.cc:407] Connection header received
0811 04:36:53.794714 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.794718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.794784 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.794888 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.796331 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.797040 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.798398 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.798611 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.798703 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.801224 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.801227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.801229 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.801255 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.802838 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.803307 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.803310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.803311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.803388 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.803692 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.803695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.803696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.803875 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.803951 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.804563 (+ 612us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.804674 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260811 04:36:53.805874 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.792428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38261 (local address 127.2.238.131:40149)
0811 04:36:53.792885 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:53.792930 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:53.794402 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:53.795093 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:53.795101 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:53.795559 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:53.796178 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.796193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.797182 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.797186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.798245 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:53.798256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.798754 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.798757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:53.798928 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.799484 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:53.799501 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:53.801062 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:53.802940 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.802945 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.802959 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.803217 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.803470 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:53.803472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:53.803474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:53.803608 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:53.803965 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:53.803970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:53.804301 (+ 331us) client_negotiation.cc:769] Sending connection context
0811 04:36:53.804478 (+ 177us) client_negotiation.cc:240] Negotiation successful
0811 04:36:53.804808 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260811 04:36:53.807058 9579 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38261
I20260811 04:36:53.807305 9579 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:53.807780 9579 heartbeater.cc:511] Master 127.2.238.190:38261 requested a full tablet report, sending...
I20260811 04:36:53.809185 9093 ts_manager.cc:194] Registered new tserver with Master: 1951f031ee93470884ee8a5e513be253 (127.2.238.131:36063)
I20260811 04:36:53.809876 9093 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:40149
I20260811 04:36:53.821556 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/test-admin.keytab.
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.838439 3002 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260811 04:36:53.841641 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:53.839981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.2.238.129:44989)
0811 04:36:53.840365 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.840381 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.840406 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:36:53.840458 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.840464 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.840655 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.840780 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.841398 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39044: BlockingRecv error: recv got EOF from 127.0.0.1:39044 (error 108)
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":163,"threads_started":1}
W20260811 04:36:53.841652 3002 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc' of type FILE
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.842798 3002 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.843740 3002 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:53.853978 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.844667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.2.238.129:44989)
0811 04:36:53.844811 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.844816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.844956 (+ 140us) server_negotiation.cc:407] Connection header received
0811 04:36:53.845123 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.845126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.845169 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.845253 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.846009 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.847068 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.847884 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.848266 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.848349 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.850622 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.850632 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.850648 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.850673 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.852091 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.852508 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.852513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.852515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.852592 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.852830 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.852832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.852833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.853130 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.853362 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.853570 (+ 208us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.853759 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 04:36:53.857914 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:53.856872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39062 (local address 127.2.238.129:44989)
0811 04:36:53.857114 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.857120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.857156 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:36:53.857247 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.857250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.857293 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.857355 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.857749 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39062: BlockingRecv error: recv got EOF from 127.0.0.1:39062 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260811 04:36:53.858079 3002 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc' of type FILE
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.859249 3002 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.860535 3002 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:53.870398 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.861710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.2.238.129:44989)
0811 04:36:53.861874 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.861879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.861988 (+ 109us) server_negotiation.cc:407] Connection header received
0811 04:36:53.862173 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.862177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.862229 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.862320 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.862904 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.863460 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.864298 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.864654 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.864760 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.867388 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.867391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.867393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.867419 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.868693 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.869088 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.869091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.869092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.869165 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.869394 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.869396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.869398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.869571 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.869685 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.870053 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.870196 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260811 04:36:53.873318 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:36:53.872259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.2.238.129:44989)
0811 04:36:53.872424 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.872430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.872648 (+ 218us) server_negotiation.cc:407] Connection header received
0811 04:36:53.872739 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.872741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.872781 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.872838 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.873204 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingRecv error: recv got EOF from 127.0.0.1:39080 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 04:36:53.873695 3002 init.cc:244] Error while opening credential cache '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786422982503306-3002-0/krb5kdc/krb5cc)
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.874825 3002 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:53.875747 3002 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:36:53.884526 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:53.876729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.2.238.129:44989)
0811 04:36:53.876860 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:53.876864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:53.877002 (+ 138us) server_negotiation.cc:407] Connection header received
0811 04:36:53.877097 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:53.877099 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:53.877142 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:53.877208 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:53.877681 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.878246 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.878997 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:53.879200 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:53.879282 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:53.881542 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:53.881545 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:53.881547 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:53.881571 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:53.882847 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.883256 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.883259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.883260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.883352 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:53.883690 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:53.883693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:53.883694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:53.883867 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:53.883966 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:53.884195 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 04:36:53.884325 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:53.885648 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9158
I20260811 04:36:53.894062 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9304
I20260811 04:36:53.903923 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9443
I20260811 04:36:53.910979 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9059
2026-08-11T04:36:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2292 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9601](info): set up 2 sockets
Aug 11 04:36:53 dist-test-slave-0kls krb5kdc[9601](info): commencing operation
krb5kdc: starting...
W20260811 04:36:54.999078 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:36:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:36:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab.
Entry for principal oryx/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab.
I20260811 04:36:55.129281 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:35291
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab
--principal=oryx/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:35291
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:55.246156 9615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:55.246495 9615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:55.246580 9615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:55.250598 9615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:55.250710 9615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:55.250748 9615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:55.250787 9615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:55.250820 9615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:55.256373 9615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:35291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab
--principal=oryx/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:35291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:55.257655 9615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:55.258817 9615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:55.267863 9623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:55.267863 9620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:55.267890 9621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:55.268015 9615 server_base.cc:1034] running on GCE node
I20260811 04:36:55.268843 9615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:55.270123 9615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:55.271350 9615 hybrid_clock.cc:648] HybridClock initialized: now 1786423015271318 us; error 42 us; skew 500 ppm
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:55.274243 9615 init.cc:377] Logged in from keytab as oryx/127.2.238.190@KRBTEST.COM (short username oryx)
I20260811 04:36:55.275305 9615 webserver.cc:460] Webserver started at http://127.2.238.190:41461/ using document root <none> and password file <none>
I20260811 04:36:55.275924 9615 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:55.276024 9615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:55.276286 9615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:55.278893 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "a21e91154bc8434ea619e020161ae1c5"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "06b8955b9e9f257c8cae12d50d7971d2"
server_key_iv: "2ebd7742fbaefd7764304b031b30e3fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.279597 9615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "a21e91154bc8434ea619e020161ae1c5"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "06b8955b9e9f257c8cae12d50d7971d2"
server_key_iv: "2ebd7742fbaefd7764304b031b30e3fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.283787 9615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:55.286445 9630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.287701 9615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:55.287880 9615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "a21e91154bc8434ea619e020161ae1c5"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "06b8955b9e9f257c8cae12d50d7971d2"
server_key_iv: "2ebd7742fbaefd7764304b031b30e3fc"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.288005 9615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:55.308754 9615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:55.312263 9615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:55.312530 9615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:55.320218 9615 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:35291
I20260811 04:36:55.320528 9686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:35291 every 8 connection(s)
I20260811 04:36:55.321321 9615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:36:55.324265 9687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:55.326889 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9615
I20260811 04:36:55.327039 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:36:55.327353 3002 external_mini_cluster.cc:1428] Setting key 2c92bf71b4b50f56a68438ff27535bf8
I20260811 04:36:55.330754 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Bootstrap starting.
I20260811 04:36:55.333451 9687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:55.334280 9687 log.cc:824] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:55.336747 9687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: No bootstrap required, opened a new log
I20260811 04:36:55.340988 9687 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } }
I20260811 04:36:55.341367 9687 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:55.341457 9687 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21e91154bc8434ea619e020161ae1c5, State: Initialized, Role: FOLLOWER
I20260811 04:36:55.341964 9687 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } }
I20260811 04:36:55.342119 9687 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:36:55.342207 9687 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:36:55.342335 9687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:55.342471 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:55.328792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.2.238.190:35291)
0811 04:36:55.329221 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:55.329237 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:55.329272 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:36:55.330019 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:55.330038 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:55.330389 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:55.330661 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:55.331618 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.332621 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.333826 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.334281 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.334511 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:55.337374 (+ 2863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:55.337379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:55.337393 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:55.337421 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:55.339237 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.339748 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.339753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.339758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.339894 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.340196 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.340198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.340200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.340599 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:55.340748 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:55.341074 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 04:36:55.341396 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":148,"threads_started":1}
I20260811 04:36:55.343348 9687 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } }
I20260811 04:36:55.343726 9687 leader_election.cc:302] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21e91154bc8434ea619e020161ae1c5; no voters:
I20260811 04:36:55.344079 9687 leader_election.cc:288] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:36:55.344195 9692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:55.344485 9692 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 LEADER]: Becoming Leader. State: Replica: a21e91154bc8434ea619e020161ae1c5, State: Running, Role: LEADER
I20260811 04:36:55.345108 9692 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } }
I20260811 04:36:55.345404 9687 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:55.346839 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e91154bc8434ea619e020161ae1c5. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } }
I20260811 04:36:55.346974 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:55.347262 9693 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } }
I20260811 04:36:55.347375 9693 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
W20260811 04:36:55.350353 9707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:36:55.350431 9707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:36:55.350487 9703 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:55.351341 9703 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:55.356488 9703 catalog_manager.cc:1347] Generated new cluster ID: 6fa29f39b6ff431882770b9e329403e9
I20260811 04:36:55.356550 9703 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:55.372500 9703 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:36:55.373616 9703 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:55.387501 9703 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Generated new TSK 0
I20260811 04:36:55.388309 9703 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
I20260811 04:36:55.420256 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:55.541872 9713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:55.542166 9713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:55.542249 9713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:55.546460 9713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:55.546545 9713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:55.546651 9713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:55.551913 9713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:55.552906 9713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:55.553951 9713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:36:55.563580 9713 server_base.cc:1034] running on GCE node
W20260811 04:36:55.563532 9721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:55.563643 9718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:55.563532 9719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:55.564193 9713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:55.564810 9713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:55.566009 9713 hybrid_clock.cc:648] HybridClock initialized: now 1786423015565996 us; error 23 us; skew 500 ppm
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:55.568643 9713 init.cc:377] Logged in from keytab as oryx/127.2.238.129@KRBTEST.COM (short username oryx)
I20260811 04:36:55.569541 9713 webserver.cc:460] Webserver started at http://127.2.238.129:44691/ using document root <none> and password file <none>
I20260811 04:36:55.570176 9713 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:55.570269 9713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:55.570508 9713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:55.572504 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "b9aec8dc0349403ca768eb48393bea0b"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "a26c8d86d8cc789dc5beb31851b72d85"
server_key_iv: "3a7c0d1c5398a1f0ad5ba7e1c476dce5"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.573134 9713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "b9aec8dc0349403ca768eb48393bea0b"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "a26c8d86d8cc789dc5beb31851b72d85"
server_key_iv: "3a7c0d1c5398a1f0ad5ba7e1c476dce5"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.577003 9713 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:36:55.579510 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.580708 9713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:55.580854 9713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "b9aec8dc0349403ca768eb48393bea0b"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "a26c8d86d8cc789dc5beb31851b72d85"
server_key_iv: "3a7c0d1c5398a1f0ad5ba7e1c476dce5"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.580971 9713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:55.607008 9713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:55.610019 9713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:55.610230 9713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:55.610857 9713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:55.611720 9713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:55.611791 9713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.611864 9713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:55.611912 9713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.620759 9713 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40797
I20260811 04:36:55.620786 9845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40797 every 8 connection(s)
I20260811 04:36:55.621681 9713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:55.629454 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9713
I20260811 04:36:55.629595 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:36:55.629981 3002 external_mini_cluster.cc:1428] Setting key 8846a7acf2e652b7ef9499327b9d07af
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:55.635286 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:55.623553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:46593 (local address 127.2.238.190:35291)
0811 04:36:55.623708 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:55.623713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:55.624499 (+ 786us) server_negotiation.cc:407] Connection header received
0811 04:36:55.625744 (+ 1245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:55.625747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:55.625848 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:55.625919 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:55.627072 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.627583 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.628665 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.628865 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.628933 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:55.631551 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:55.631555 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:55.631557 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:55.631580 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:55.633198 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.633689 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.633693 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.633695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.633830 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.634177 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.634181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.634183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.634372 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:55.634477 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:55.634962 (+ 485us) server_negotiation.cc:299] Negotiation successful
0811 04:36:55.635087 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:36:55.636121 9848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:55.623891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:46593)
0811 04:36:55.624324 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:55.624367 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:55.625550 (+ 1183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:55.626078 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:55.626084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:55.626477 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:55.626942 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:55.626955 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.627698 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.627700 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:55.628546 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:55.628554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.628986 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.628988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:55.629146 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:55.629894 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:55.629911 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:55.631404 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:55.633320 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:55.633325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:55.633337 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:55.633581 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:55.633923 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:55.633926 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:55.633928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:55.634080 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:55.634490 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:55.634498 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:55.634727 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 04:36:55.634922 (+ 195us) client_negotiation.cc:240] Negotiation successful
0811 04:36:55.635258 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":105,"threads_started":1}
I20260811 04:36:55.637573 9846 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:55.637854 9846 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:55.638370 9846 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:55.640154 9649 ts_manager.cc:194] Registered new tserver with Master: b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
WARNING: no policy specified for oryx/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:36:55.641573 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:46593
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
I20260811 04:36:55.660499 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:55.775403 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:55.775650 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:55.775691 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:55.779614 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:55.779667 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:55.779721 9851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:55.784765 9851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:55.785914 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:55.786907 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:55.796429 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:55.796450 9858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:55.796489 9851 server_base.cc:1034] running on GCE node
W20260811 04:36:55.796447 9860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:55.797058 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:55.797660 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:55.798851 9851 hybrid_clock.cc:648] HybridClock initialized: now 1786423015798835 us; error 26 us; skew 500 ppm
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:55.801476 9851 init.cc:377] Logged in from keytab as oryx/127.2.238.130@KRBTEST.COM (short username oryx)
I20260811 04:36:55.802347 9851 webserver.cc:460] Webserver started at http://127.2.238.130:32833/ using document root <none> and password file <none>
I20260811 04:36:55.802901 9851 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:55.802984 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:55.803205 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:55.805099 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "4147a38de662745e68fa99e169c00045"
server_key_iv: "156ca51a0ef642938991fbe5d9d0cb9d"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.805655 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "4147a38de662745e68fa99e169c00045"
server_key_iv: "156ca51a0ef642938991fbe5d9d0cb9d"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.809271 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:36:55.812031 9867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.813269 9851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:36:55.813405 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "4147a38de662745e68fa99e169c00045"
server_key_iv: "156ca51a0ef642938991fbe5d9d0cb9d"
server_key_version: "encryptionkey@0"
I20260811 04:36:55.813512 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:55.835384 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:55.838265 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:55.838469 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:55.839022 9851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:55.839811 9851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:55.839908 9851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.839982 9851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:55.840035 9851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:55.849304 9851 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:44123
I20260811 04:36:55.849323 9983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:44123 every 8 connection(s)
I20260811 04:36:55.850250 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:55.858366 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9851
I20260811 04:36:55.858456 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:36:55.858731 3002 external_mini_cluster.cc:1428] Setting key 6b6d89a7cc485e7442d0b3cb43ea2a6f
Aug 11 04:36:55 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:55.868306 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:55.851901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:59531 (local address 127.2.238.190:35291)
0811 04:36:55.852013 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:55.852018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:55.852743 (+ 725us) server_negotiation.cc:407] Connection header received
0811 04:36:55.854036 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:55.854039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:55.854101 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:55.854204 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:55.855278 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.855773 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.856843 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.857044 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.857142 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:55.859622 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:55.859625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:55.859627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:55.859652 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:55.866135 (+ 6483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.866644 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.866651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.866654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.866765 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:55.867087 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:55.867091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:55.867092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:55.867362 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:55.867488 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:55.867957 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 04:36:55.868092 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 04:36:55.869005 9987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:55.852192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:59531)
0811 04:36:55.852596 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:55.852633 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:55.853827 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:55.854374 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:55.854381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:55.854705 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:55.855132 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:55.855144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.855904 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.855907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:55.856722 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:55.856731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:55.857190 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:55.857194 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:55.857346 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:55.858001 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:55.858017 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:55.859451 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:55.866265 (+ 6814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:55.866269 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:55.866282 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:55.866533 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:55.866867 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:55.866869 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:55.866870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:55.866995 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:55.867482 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:55.867486 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:55.867715 (+ 229us) client_negotiation.cc:769] Sending connection context
0811 04:36:55.867861 (+ 146us) client_negotiation.cc:240] Negotiation successful
0811 04:36:55.868096 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for oryx/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:36:55.870255 9984 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:55.870477 9984 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:55.871042 9984 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:55.872016 9649 ts_manager.cc:194] Registered new tserver with Master: 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.131@KRBTEST.COM" created.
I20260811 04:36:55.872594 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:59531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
I20260811 04:36:55.889962 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:56.000988 9990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:56.001241 9990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:56.001308 9990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:56.008432 9990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:56.008512 9990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:56.008607 9990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:56.014003 9990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:56.015084 9990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:56.016086 9990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:56.025965 9996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.025983 9995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.025983 9998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.026032 9990 server_base.cc:1034] running on GCE node
I20260811 04:36:56.026686 9990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.027344 9990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.028532 9990 hybrid_clock.cc:648] HybridClock initialized: now 1786423016028517 us; error 26 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.031242 9990 init.cc:377] Logged in from keytab as oryx/127.2.238.131@KRBTEST.COM (short username oryx)
I20260811 04:36:56.032130 9990 webserver.cc:460] Webserver started at http://127.2.238.131:34307/ using document root <none> and password file <none>
I20260811 04:36:56.032702 9990 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.032792 9990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.033058 9990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:56.035216 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "977af38b0ccd47429939f34995f7cdeb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "62e426f2a863167867f424078946b9d2"
server_key_iv: "5a8a7f17de214706b22c2999af01f8e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.035789 9990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "977af38b0ccd47429939f34995f7cdeb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "62e426f2a863167867f424078946b9d2"
server_key_iv: "5a8a7f17de214706b22c2999af01f8e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.039531 9990 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260811 04:36:56.041954 10006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.043053 9990 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:36:56.043181 9990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "977af38b0ccd47429939f34995f7cdeb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "62e426f2a863167867f424078946b9d2"
server_key_iv: "5a8a7f17de214706b22c2999af01f8e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.043316 9990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.067824 9990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.070808 9990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:56.071015 9990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.071620 9990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:56.072472 9990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:56.072520 9990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.072587 9990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:56.072621 9990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.082698 9990 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:37629
I20260811 04:36:56.082718 10131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:37629 every 8 connection(s)
I20260811 04:36:56.083601 9990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:56.087387 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 9990
I20260811 04:36:56.087505 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:36:56.087831 3002 external_mini_cluster.cc:1428] Setting key 48ce0cd882493c524dde0e2da36c93f8
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:56.097230 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.085492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:54303 (local address 127.2.238.190:35291)
0811 04:36:56.085665 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.085671 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.086796 (+ 1125us) server_negotiation.cc:407] Connection header received
0811 04:36:56.087977 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.087980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.088038 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.088113 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.089530 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.090199 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.091261 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.091478 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.091571 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.093983 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.093986 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.093989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.094022 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.095218 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.095747 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.095751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.095752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.095834 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.096187 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.096192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.096194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.096378 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.096485 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.096952 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.097069 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 04:36:56.098019 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.085975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:54303)
0811 04:36:56.086645 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.086685 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.087706 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.088329 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.088335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.088764 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.089391 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.089411 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.090308 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.090311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.091161 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.091169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.091645 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.091651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.091798 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.092325 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.092341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.093775 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.095345 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.095351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.095364 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.095632 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.095927 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.095930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.095932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.096090 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.096484 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.096491 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.096717 (+ 226us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.096898 (+ 181us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.097189 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":116,"threads_started":1}
I20260811 04:36:56.099056 10132 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:56.099259 10132 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:56.099668 10132 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:56.100715 9649 ts_manager.cc:194] Registered new tserver with Master: 977af38b0ccd47429939f34995f7cdeb (127.2.238.131:37629)
I20260811 04:36:56.101397 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:54303
I20260811 04:36:56.101902 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.189.keytab.
Entry for principal oryx/127.2.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.189.keytab.
I20260811 04:36:56.128698 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:43273
--webserver_interface=127.2.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.189.keytab
--principal=oryx/127.2.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:35291,127.2.238.189:43273
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:36:56.238572 10137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:56.238795 10137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:56.238832 10137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:56.242693 10137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:56.242739 10137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:56.242758 10137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:56.242774 10137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:56.242791 10137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:56.247886 10137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:35291,127.2.238.189:43273
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.189.keytab
--principal=oryx/127.2.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:43273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
W20260811 04:36:56.248042 10137 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.238.190:35291,127.2.238.189:43273'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260811 04:36:56.248783 10137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:56.249676 10137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:36:56.257623 10137 server_base.cc:1034] running on GCE node
W20260811 04:36:56.257565 10147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.257593 10144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.257575 10145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.258416 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.259400 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.260560 10137 hybrid_clock.cc:648] HybridClock initialized: now 1786423016260541 us; error 28 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.263334 10137 init.cc:377] Logged in from keytab as oryx/127.2.238.189@KRBTEST.COM (short username oryx)
I20260811 04:36:56.264173 10137 webserver.cc:460] Webserver started at http://127.2.238.189:36013/ using document root <none> and password file <none>
I20260811 04:36:56.264711 10137 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.264796 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.265038 10137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:56.267038 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/instance:
uuid: "33d56bc0cbf94c25a30c5382b53e5360"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "ee680dcf3947eb5bb9c4de2dadbf6118"
server_key_iv: "1e6616046aa60d6b389a4d7917a3712d"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.267601 10137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal/instance:
uuid: "33d56bc0cbf94c25a30c5382b53e5360"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "ee680dcf3947eb5bb9c4de2dadbf6118"
server_key_iv: "1e6616046aa60d6b389a4d7917a3712d"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.271083 10137 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:36:56.273401 10154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.274648 10137 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:56.274787 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "33d56bc0cbf94c25a30c5382b53e5360"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "ee680dcf3947eb5bb9c4de2dadbf6118"
server_key_iv: "1e6616046aa60d6b389a4d7917a3712d"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.274891 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.299127 10137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.301884 10137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:56.302106 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.308729 10137 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.189:43273
I20260811 04:36:56.309005 10215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.189:43273 every 8 connection(s)
I20260811 04:36:56.309710 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
I20260811 04:36:56.312480 10216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:56.314834 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 10137
I20260811 04:36:56.316766 10216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.189@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:56.329941 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.319241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41678 (local address 127.2.238.190:35291)
0811 04:36:56.319382 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.319387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.319702 (+ 315us) server_negotiation.cc:407] Connection header received
0811 04:36:56.319897 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.319900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.319955 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.320035 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.320962 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.321452 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.322879 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.323166 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.323271 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.326156 (+ 2885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.326161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.326168 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.326193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.327787 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.328386 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.328389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.328391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.328474 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.328796 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.328799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.328801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.329073 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.329172 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.329579 (+ 407us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.329712 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:36:56.330927 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.316380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.2.238.189:43273)
0811 04:36:56.316743 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.316758 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.316792 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:36:56.317440 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.317457 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.317930 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.318303 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.319232 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.320784 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.321952 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.322574 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.322742 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.326060 (+ 3318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.326067 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.326080 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.326104 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.327971 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.328474 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.328479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.328481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.328615 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.328912 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.328915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.328916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.329330 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.329531 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.329786 (+ 255us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.330107 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":228,"spinlock_wait_cycles":5376,"thread_start_us":108,"threads_started":1}
I20260811 04:36:56.330929 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.319199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41678)
0811 04:36:56.319580 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.319618 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.319762 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.320106 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.320112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.320533 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.320768 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.320783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.321585 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.321589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.322574 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.322581 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.323430 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.323433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.323511 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.324031 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.324046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.325977 (+ 1931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.327902 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.327913 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.327923 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.328243 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.328554 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.328556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.328558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.328665 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.329202 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.329211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.329354 (+ 143us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.330084 (+ 730us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.330265 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":1,"thread_start_us":158,"threads_started":1}
I20260811 04:36:56.332388 10216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.189@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
I20260811 04:36:56.339440 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.332821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55706)
0811 04:36:56.333114 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.333127 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.333194 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.333424 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.333428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.333602 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.333708 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.333715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.334476 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.334479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.335001 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.335010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.335378 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.335380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.335443 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.335783 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.335793 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.336958 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.338341 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.338343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.338344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.338514 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.338731 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.338733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.338734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.338799 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.339111 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.339113 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.339175 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.339246 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.339326 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260811 04:36:56.339495 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.333088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55706 (local address 127.2.238.189:43273)
0811 04:36:56.333211 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.333215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.333228 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:56.333258 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.333259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.333290 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.333352 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.333857 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.334354 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.335095 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.335291 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.335357 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.337075 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.337078 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.337080 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.337100 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.338249 (+ 1149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.338593 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.338595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.338596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.338658 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.338870 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.338872 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.338873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.339023 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.339105 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.339299 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.339375 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 04:36:56.342581 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Bootstrap starting.
I20260811 04:36:56.344724 10216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:56.345400 10216 log.cc:824] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:56.347323 10216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: No bootstrap required, opened a new log
I20260811 04:36:56.350404 10216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } }
I20260811 04:36:56.350616 10216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:56.350701 10216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d56bc0cbf94c25a30c5382b53e5360, State: Initialized, Role: FOLLOWER
I20260811 04:36:56.351137 10216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } }
I20260811 04:36:56.351585 10222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } } }
I20260811 04:36:56.351691 10222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:56.352291 10216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: configured and running, proceeding with master startup.
W20260811 04:36:56.356870 10233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:36:56.356972 10233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:36:56.358745 10137 master_runner.cc:428] Detected that this master 33d56bc0cbf94c25a30c5382b53e5360 is joining an existing cluster
I20260811 04:36:56.358806 10137 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.238.189:43273
I20260811 04:36:56.369455 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.361070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41690 (local address 127.2.238.190:35291)
0811 04:36:56.361185 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.361189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.361511 (+ 322us) server_negotiation.cc:407] Connection header received
0811 04:36:56.361646 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.361649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.361697 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.361775 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.362379 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.363314 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364376 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.364643 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364749 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.366377 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.366381 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.366383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.366410 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.367853 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368273 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.368277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.368278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.368359 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368605 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.368611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.368615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.368825 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.368938 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.369197 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.369306 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:36:56.369467 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.361416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55716 (local address 127.2.238.189:43273)
0811 04:36:56.361525 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.361529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.361842 (+ 313us) server_negotiation.cc:407] Connection header received
0811 04:36:56.361985 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.361988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.362028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.362109 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.362556 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.363073 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364156 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.364529 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364641 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.366027 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.366030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.366033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.366083 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.367875 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368348 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.368351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.368352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.368421 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368656 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.368658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.368660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.368861 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.368954 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.369219 (+ 265us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.369348 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":51}
I20260811 04:36:56.369617 10241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.360977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55716)
0811 04:36:56.361732 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.361764 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.361891 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.362141 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.362147 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.362349 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.362456 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.362461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.363189 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.363192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.364036 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.364045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.364658 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.364773 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.365309 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.365320 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.365897 (+ 577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.367986 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.367988 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.367990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.368273 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.368491 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.368496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.368584 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.368983 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.368985 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.369081 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.369268 (+ 187us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.369418 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":58,"spinlock_wait_cycles":2304,"thread_start_us":76,"threads_started":1}
I20260811 04:36:56.369839 10240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.360977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41690)
0811 04:36:56.361419 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.361431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.361546 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.361872 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.361875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.362112 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.362254 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.362262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.363423 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.363426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.364239 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.364245 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.364898 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.364901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.365010 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.365488 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.365502 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.365903 (+ 401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.367968 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.367972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.367974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.368179 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.368432 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.368434 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.368436 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.368513 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.368929 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.368932 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.369035 (+ 103us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.369146 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.369647 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"spinlock_wait_cycles":1408,"thread_start_us":118,"threads_started":1}
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.190@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
I20260811 04:36:56.382774 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.374410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55730 (local address 127.2.238.189:43273)
0811 04:36:56.374514 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.374518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.374820 (+ 302us) server_negotiation.cc:407] Connection header received
0811 04:36:56.375043 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.375046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.375097 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.375178 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.375797 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.376261 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.377083 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.377290 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.377400 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.379747 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.379751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.379753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.379775 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.381103 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.381565 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.381568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.381570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.381644 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.381958 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.381961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.381962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.382113 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.382211 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.382531 (+ 320us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.382632 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:36:56.382788 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.374382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55730)
0811 04:36:56.374707 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.374747 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.374888 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.375213 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.375219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.375560 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.375663 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.375669 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.376369 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.376373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.376964 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.376973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.377405 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.377408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.377541 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.378056 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.378068 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.379605 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.381229 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.381247 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.381252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.381461 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.381728 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.381730 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.381732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.381858 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.382215 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.382223 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.382339 (+ 116us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.382481 (+ 142us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.382648 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260811 04:36:56.384712 9649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.238.189:43273: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360"
member_type: NON_VOTER
last_known_addr {
host: "127.2.238.189"
port: 43273
}
attrs {
promote: true
}
}
dest_uuid: "a21e91154bc8434ea619e020161ae1c5"
cas_config_opid_index: -1
I20260811 04:36:56.385419 9649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } }
I20260811 04:36:56.387372 10244 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } }
I20260811 04:36:56.387928 10186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:56.389031 10244 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.238.189:43273
I20260811 04:36:56.389242 10186 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260811 04:36:56.389480 10137 master.cc:537] Master@127.2.238.189:43273 shutting down...
I20260811 04:36:56.389608 10245 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } }
I20260811 04:36:56.389890 10244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:56.390127 10245 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
W20260811 04:36:56.391662 9632 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260811 04:36:56.392647 9632 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 -> Peer 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189:43273): Couldn't send request to peer 33d56bc0cbf94c25a30c5382b53e5360. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.2.238.188@KRBTEST.COM; defaulting to no policy
I20260811 04:36:56.396597 10137 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:56.396862 10137 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:56.396972 10137 tablet_replica.cc:330] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.188@KRBTEST.COM" created.
I20260811 04:36:56.402611 10137 master.cc:552] Master@127.2.238.189:43273 shutdown complete.
I20260811 04:36:56.402675 10137 master_runner.cc:305] Clearing existing system tablet
I20260811 04:36:56.402884 10137 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.402973 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.404054 10137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.404875 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.405061 10137 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.405176 10137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.405217 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "33d56bc0cbf94c25a30c5382b53e5360"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "ee680dcf3947eb5bb9c4de2dadbf6118"
server_key_iv: "1e6616046aa60d6b389a4d7917a3712d"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.406865 10137 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:56.408671 10137 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 04:36:56.408782 10137 log.cc:1196] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260811 04:36:56.409042 10137 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Deleting consensus metadata
I20260811 04:36:56.410055 10137 master_runner.cc:315] Copying system tablet from 127.2.238.190:35291
I20260811 04:36:56.411178 10137 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: tablet copy: Beginning tablet copy session from remote peer at address 127.2.238.190:35291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.188.keytab.
Entry for principal oryx/127.2.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.188.keytab.
I20260811 04:36:56.416687 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:35905
--webserver_interface=127.2.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.188.keytab
--principal=oryx/127.2.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:36:56.420253 10257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.411739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41704)
0811 04:36:56.412160 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.412171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.412302 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.412610 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.412613 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.412789 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.412896 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.412901 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.413705 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.413728 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.414472 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.414482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.415058 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.415061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.415245 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.416278 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.416294 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.416701 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.418719 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.418723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.418725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.418927 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.419207 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.419210 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.419212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.419301 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.419667 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.419669 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.419776 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.419917 (+ 141us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.420052 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":222,"threads_started":1}
I20260811 04:36:56.420251 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.411817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41704 (local address 127.2.238.190:35291)
0811 04:36:56.411988 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.411993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.412245 (+ 252us) server_negotiation.cc:407] Connection header received
0811 04:36:56.412460 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.412462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.412510 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.412595 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.413008 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.413605 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.414591 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.414941 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.415053 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.416859 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.416865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.416870 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.416901 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.418608 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.419031 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.419035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.419036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.419115 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.419393 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.419397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.419399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.419571 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.419677 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.419976 (+ 299us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.420085 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:36:56.420789 9670 tablet_copy_service.cc:140] P a21e91154bc8434ea619e020161ae1c5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 33d56bc0cbf94c25a30c5382b53e5360 ({username='oryx', principal='oryx/127.2.238.189@KRBTEST.COM'} at 127.0.0.1:41704)
I20260811 04:36:56.420966 9670 tablet_copy_service.cc:161] P a21e91154bc8434ea619e020161ae1c5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 33d56bc0cbf94c25a30c5382b53e5360 at {username='oryx', principal='oryx/127.2.238.189@KRBTEST.COM'} at 127.0.0.1:41704: session id = 33d56bc0cbf94c25a30c5382b53e5360-00000000000000000000000000000000
I20260811 04:36:56.422605 9670 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 04:36:56.424072 10137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 04:36:56.425767 10137 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: tablet copy: Starting download of 0 data blocks...
I20260811 04:36:56.425982 10137 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: tablet copy: Starting download of 1 WAL segments...
I20260811 04:36:56.427572 10137 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 04:36:56.428987 9670 tablet_copy_service.cc:342] P a21e91154bc8434ea619e020161ae1c5: Request end of tablet copy session 33d56bc0cbf94c25a30c5382b53e5360-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.238.189@KRBTEST.COM'} at 127.0.0.1:41704
I20260811 04:36:56.429103 9670 tablet_copy_service.cc:433] P a21e91154bc8434ea619e020161ae1c5: ending tablet copy session 33d56bc0cbf94c25a30c5382b53e5360-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 33d56bc0cbf94c25a30c5382b53e5360
I20260811 04:36:56.436493 10137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:56.438076 10261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.438140 10260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.438576 10137 server_base.cc:1034] running on GCE node
W20260811 04:36:56.441896 10263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.442121 10137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.442435 10137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.443572 10137 hybrid_clock.cc:648] HybridClock initialized: now 1786423016443570 us; error 53 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.445079 10137 init.cc:377] Logged in from keytab as oryx/127.2.238.189@KRBTEST.COM (short username oryx)
I20260811 04:36:56.445578 10137 webserver.cc:460] Webserver started at http://127.2.238.189:36175/ using document root <none> and password file <none>
I20260811 04:36:56.445842 10137 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.445890 10137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.447189 10137 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.448175 10271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.448422 10137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.448506 10137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "33d56bc0cbf94c25a30c5382b53e5360"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "ee680dcf3947eb5bb9c4de2dadbf6118"
server_key_iv: "1e6616046aa60d6b389a4d7917a3712d"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.448587 10137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.454582 10137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.456526 10137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.462381 10137 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.189:43273
I20260811 04:36:56.462929 10137 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
I20260811 04:36:56.464385 10327 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:36:56.465173 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Bootstrap starting.
I20260811 04:36:56.470026 10326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.189:43273 every 8 connection(s)
I20260811 04:36:56.473047 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:36:56.473356 10327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: Bootstrap complete.
I20260811 04:36:56.473951 10327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } }
I20260811 04:36:56.474078 10327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33d56bc0cbf94c25a30c5382b53e5360, State: Initialized, Role: LEARNER
I20260811 04:36:56.474200 10327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } }
I20260811 04:36:56.474541 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } }
I20260811 04:36:56.474707 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: LEARNER
I20260811 04:36:56.474711 10327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:56.477702 10341 catalog_manager.cc:1259] Loaded cluster ID: 6fa29f39b6ff431882770b9e329403e9
I20260811 04:36:56.477766 10341 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: loading cluster ID for follower catalog manager: success
I20260811 04:36:56.479249 10341 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: acquiring CA information for follower catalog manager: success
I20260811 04:36:56.480340 10341 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260811 04:36:56.558878 10258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:56.559159 10258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:56.559234 10258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:56.563407 10258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:36:56.563477 10258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:56.563514 10258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:56.563568 10258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:36:56.563602 10258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:36:56.568725 10258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.188.keytab
--principal=oryx/127.2.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:56.569721 10258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:56.570758 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:56.579236 10349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.579222 10347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.579211 10346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.579558 10258 server_base.cc:1034] running on GCE node
I20260811 04:36:56.580147 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.581212 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.582386 10258 hybrid_clock.cc:648] HybridClock initialized: now 1786423016582376 us; error 29 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.585028 10258 init.cc:377] Logged in from keytab as oryx/127.2.238.188@KRBTEST.COM (short username oryx)
I20260811 04:36:56.585935 10258 webserver.cc:460] Webserver started at http://127.2.238.188:46265/ using document root <none> and password file <none>
I20260811 04:36:56.586506 10258 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.586606 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.586854 10258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:36:56.588864 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/instance:
uuid: "8f20a8835d7546818bb1c212a5abdcdb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "9066c2986d1e53282ddb0f3088bcc839"
server_key_iv: "d4479c987fde78b9fc302fac05e61f9c"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.589397 10258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal/instance:
uuid: "8f20a8835d7546818bb1c212a5abdcdb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "9066c2986d1e53282ddb0f3088bcc839"
server_key_iv: "d4479c987fde78b9fc302fac05e61f9c"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.593042 10258 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260811 04:36:56.595350 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.596499 10258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:36:56.596640 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "8f20a8835d7546818bb1c212a5abdcdb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "9066c2986d1e53282ddb0f3088bcc839"
server_key_iv: "d4479c987fde78b9fc302fac05e61f9c"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.596747 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.605772 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.608474 10258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:56.608675 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.616021 10258 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.188:35905
I20260811 04:36:56.616288 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.188:35905 every 8 connection(s)
I20260811 04:36:56.616997 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
I20260811 04:36:56.619818 10414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:56.623538 10414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:36:56.624249 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 10258
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.188@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
I20260811 04:36:56.636467 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.625495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41720 (local address 127.2.238.190:35291)
0811 04:36:56.625638 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.625644 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.626328 (+ 684us) server_negotiation.cc:407] Connection header received
0811 04:36:56.627007 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.627011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.627056 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.627143 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.628433 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.628973 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.630062 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.630270 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.630347 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.632903 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.632910 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.632915 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.632943 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.634372 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.634918 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.634922 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.634923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.635024 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.635434 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.635438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.635439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.635619 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.635720 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.636181 (+ 461us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.636299 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:36:56.637364 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.625681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41720)
0811 04:36:56.626182 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.626228 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.626811 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.627561 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.627565 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.627915 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.628279 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.628293 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.629098 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.629102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.629937 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.629947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.630403 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.630409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.630579 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.631273 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.631288 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.632741 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.634471 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.634482 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.634492 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.634806 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.635111 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.635114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.635116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.635339 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.635719 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.635725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.635961 (+ 236us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.636140 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.636433 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":72,"spinlock_wait_cycles":21504,"thread_start_us":212,"threads_started":1}
I20260811 04:36:56.637364 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.625681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.2.238.188:35905)
0811 04:36:56.626094 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.626107 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.626139 (+ 32us) server_negotiation.cc:407] Connection header received
0811 04:36:56.626790 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.626806 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.627023 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.627182 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.627906 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.629135 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.630025 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.630449 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.630577 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.633458 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.633463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.633472 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.633491 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.635150 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.635585 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.635590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.635591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.635660 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.635950 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.635953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.635955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.636172 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.636383 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.636581 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.636715 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":285,"thread_start_us":166,"threads_started":1}
I20260811 04:36:56.638722 10414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.188@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
I20260811 04:36:56.644182 9846 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
I20260811 04:36:56.647224 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.639227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55742)
0811 04:36:56.639619 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.639634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.639716 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.640006 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.640009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.640172 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.640264 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.640270 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.641160 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.641165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.641691 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.641698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.642204 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.642208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.642303 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.642712 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.642726 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.643971 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.645927 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.645930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.645931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.646173 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.646427 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.646429 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.646431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.646513 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.646810 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.646812 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.646884 (+ 72us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.646967 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.647072 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298}
I20260811 04:36:56.647302 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.639369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.2.238.189:43273)
0811 04:36:56.639626 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.639643 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.639679 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:36:56.639830 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.639833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.639900 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.639995 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.640384 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.641050 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.641840 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.642073 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.642206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.644109 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.644113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.644116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.644139 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.645823 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.646257 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.646260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.646261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.646333 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.646572 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.646574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.646575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.646725 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.646799 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.647004 (+ 205us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.647108 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":105,"threads_started":1}
I20260811 04:36:56.648126 10414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.188@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
I20260811 04:36:56.655117 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.648581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46444)
0811 04:36:56.648700 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.648713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.648788 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.649000 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.649002 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.649111 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.649173 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.649177 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.649926 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.649928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.650404 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.650411 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.650827 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.650829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.650885 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.651323 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.651335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.652391 (+ 1056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.653731 (+ 1340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.653733 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.653734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.653969 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.654200 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.654202 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.654203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.654273 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.654640 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.654644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.654719 (+ 75us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.654824 (+ 105us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.654949 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260811 04:36:56.655141 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.648723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.2.238.188:35905)
0811 04:36:56.648827 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.648831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.648845 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:36:56.648887 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.648889 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.648916 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.648972 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.649258 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.649717 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.650497 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.650719 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.650802 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.652497 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.652500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.652502 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.652519 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.653636 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.654050 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.654052 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.654053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.654121 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.654377 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.654380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.654382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.654542 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.654640 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.654870 (+ 230us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.655020 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":39}
I20260811 04:36:56.658339 10414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Bootstrap starting.
I20260811 04:36:56.660611 10414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:56.661360 10414 log.cc:824] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:56.663280 10414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: No bootstrap required, opened a new log
I20260811 04:36:56.666168 10414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } }
I20260811 04:36:56.666388 10414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:56.666469 10414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f20a8835d7546818bb1c212a5abdcdb, State: Initialized, Role: FOLLOWER
I20260811 04:36:56.666908 10414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } }
I20260811 04:36:56.667428 10420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } } }
I20260811 04:36:56.667527 10420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:56.668147 10414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: configured and running, proceeding with master startup.
W20260811 04:36:56.672910 10431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:36:56.673017 10431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:36:56.676165 10258 master_runner.cc:428] Detected that this master 8f20a8835d7546818bb1c212a5abdcdb is joining an existing cluster
I20260811 04:36:56.676227 10258 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.238.188:35905
I20260811 04:36:56.690956 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9713
I20260811 04:36:56.700323 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:40797
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:35291
--webserver_port=44691
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:36:56.709061 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.678734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.2.238.189:43273)
0811 04:36:56.678912 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.678915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.679707 (+ 792us) server_negotiation.cc:407] Connection header received
0811 04:36:56.679901 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.679904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.679953 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.680005 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.680686 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681497 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.682744 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.683104 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683177 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.703262 (+ 20085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.703266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.703269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.703300 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.705159 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706735 (+ 1576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.706739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.706741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.706828 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.707421 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.707424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.707426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.707671 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.707752 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.708752 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.708892 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260811 04:36:56.709344 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.678608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.2.238.190:35291)
0811 04:36:56.678749 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.678754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.679280 (+ 526us) server_negotiation.cc:407] Connection header received
0811 04:36:56.679521 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.679524 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.679582 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.679638 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.680401 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681255 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683215 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.683548 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683606 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.703155 (+ 19549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.703161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.703165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.703202 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.705958 (+ 2756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706748 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.706755 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.706759 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.706860 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.708204 (+ 1344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.708210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.708212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.708486 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.708570 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.709023 (+ 453us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.709168 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 04:36:56.710196 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.679334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46458 (local address 127.2.238.188:35905)
0811 04:36:56.679449 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.679454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.679737 (+ 283us) server_negotiation.cc:407] Connection header received
0811 04:36:56.679897 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.679902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.679942 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.679990 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.680601 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681305 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.682616 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.682966 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.703659 (+ 20602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.703662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.703665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.703687 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.705978 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.707265 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.707268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.707269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.707330 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.708213 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.708216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.708218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.708393 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.708453 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.709625 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.710016 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":301,"server-negotiator.queue_time_us":47}
I20260811 04:36:56.710258 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.678544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41732)
0811 04:36:56.679199 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.679215 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.679354 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.679832 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.679843 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.680099 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.680258 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.680266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681391 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.681395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.683094 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.683102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.683785 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.683878 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.702339 (+ 18461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.702361 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.702947 (+ 586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.706177 (+ 3230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706182 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.706187 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.706605 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.707008 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.707014 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.707017 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.708070 (+ 1053us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.708658 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.708664 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.708803 (+ 139us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.709535 (+ 732us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.709962 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":201,"thread_start_us":100,"threads_started":1}
I20260811 04:36:56.710608 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.679291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46458)
0811 04:36:56.679667 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.679680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.679809 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.680121 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.680123 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.680397 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.680500 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.680507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681412 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.681416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.682493 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.682507 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.683193 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683198 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.683284 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.683849 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.683921 (+ 72us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.702685 (+ 18764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.706325 (+ 3640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.706334 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.707185 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.707823 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.707826 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.707828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.707908 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.708504 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.708506 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.708584 (+ 78us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.709900 (+ 1316us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.710431 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":48,"spinlock_wait_cycles":4480,"thread_start_us":176,"threads_started":1}
I20260811 04:36:56.712807 10440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.678544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55748)
0811 04:36:56.679621 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.679635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.679769 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.680160 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.680163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.680414 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.680536 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.680544 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.681658 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.681662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.682602 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.682614 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.683325 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.683328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.683421 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.684234 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.684249 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.703089 (+ 18840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.706085 (+ 2996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706088 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.706090 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.706618 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.706933 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.706936 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.706938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.707026 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.707989 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.707991 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.708085 (+ 94us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.709267 (+ 1182us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.709721 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":19,"spinlock_wait_cycles":6400,"thread_start_us":93,"threads_started":1}
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423015, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.190@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
I20260811 04:36:56.724273 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.714997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46474 (local address 127.2.238.188:35905)
0811 04:36:56.715106 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.715110 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.715240 (+ 130us) server_negotiation.cc:407] Connection header received
0811 04:36:56.715419 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.715422 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.715485 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.715571 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.716037 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.716812 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.718019 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.718384 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.718481 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.720813 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.720817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.720820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.720846 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.722538 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.723012 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.723016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.723018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.723106 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.723415 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.723418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.723419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.723622 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.723714 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.723997 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.724109 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 04:36:56.724273 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.714945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46474)
0811 04:36:56.715151 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.715168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.715270 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.715603 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.715606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.715796 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.715904 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.715910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.716935 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.716938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.717890 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.717902 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.718489 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.718493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.718603 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.719057 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.719071 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.720647 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.722657 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.722660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.722663 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.722910 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.723220 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.723223 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.723226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.723319 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.723721 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.723724 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.723824 (+ 100us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.723933 (+ 109us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.724082 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121}
I20260811 04:36:56.726162 9649 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.238.188:35905: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb"
member_type: NON_VOTER
last_known_addr {
host: "127.2.238.188"
port: 35905
}
attrs {
promote: true
}
}
dest_uuid: "a21e91154bc8434ea619e020161ae1c5"
cas_config_opid_index: 5
I20260811 04:36:56.726863 9649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } }
I20260811 04:36:56.727980 10245 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.729583 10245 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.238.188:35905
I20260811 04:36:56.729740 10244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.729871 10244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:56.730991 10258 master.cc:537] Master@127.2.238.188:35905 shutting down...
I20260811 04:36:56.731077 10387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:56.732576 10387 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 04:36:56.733233 10244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260811 04:36:56.735591 9634 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 -> Peer 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188:35905): Couldn't send request to peer 8f20a8835d7546818bb1c212a5abdcdb. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260811 04:36:56.736758 10258 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:56.737048 10258 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:56.737154 10258 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: stopping tablet replica
I20260811 04:36:56.741683 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.730034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55760 (local address 127.2.238.189:43273)
0811 04:36:56.730849 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.730854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.736337 (+ 5483us) server_negotiation.cc:407] Connection header received
0811 04:36:56.736533 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.736536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.736613 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.736678 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:56.737256 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.738106 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.740093 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.740835 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.740918 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.741078 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.741456 (+ 378us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.741538 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725}
I20260811 04:36:56.742259 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.730148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55760)
0811 04:36:56.736255 (+ 6107us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.736271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.736412 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.736910 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.736914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.736932 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:56.737025 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.737030 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.738252 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.738256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.739950 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.739962 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.741180 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.741184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.741298 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.741342 (+ 44us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.742052 (+ 710us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.742140 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6022}
I20260811 04:36:56.742489 10258 master.cc:552] Master@127.2.238.188:35905 shutdown complete.
I20260811 04:36:56.742539 10258 master_runner.cc:305] Clearing existing system tablet
I20260811 04:36:56.742790 10258 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.742841 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.742923 10299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 LEARNER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260811 04:36:56.743384 10244 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260811 04:36:56.744081 10258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260811 04:36:56.745216 10299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.747557 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e91154bc8434ea619e020161ae1c5. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.747685 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: LEARNER
I20260811 04:36:56.747769 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: NON_VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: true } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.747830 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: LEARNER
I20260811 04:36:56.748741 10244 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: attempting to promote NON_VOTER 33d56bc0cbf94c25a30c5382b53e5360 to VOTER
I20260811 04:36:56.749205 10244 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } }
I20260811 04:36:56.750682 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.750202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46482)
0811 04:36:56.750483 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.750579 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260811 04:36:56.751157 10298 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 LEARNER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260811 04:36:56.751582 10245 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260811 04:36:56.751572 10447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.751193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46494)
0811 04:36:56.751449 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.751498 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":94,"threads_started":1}
W20260811 04:36:56.752043 9634 consensus_peers.cc:487] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 -> Peer 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188:35905): Couldn't send request to peer 8f20a8835d7546818bb1c212a5abdcdb. Status: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 04:36:56.753443 10446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.753731 10299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.753976 10443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.754181 10258 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.754305 10258 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260811 04:36:56.754338 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "8f20a8835d7546818bb1c212a5abdcdb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "9066c2986d1e53282ddb0f3088bcc839"
server_key_iv: "d4479c987fde78b9fc302fac05e61f9c"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.755303 10330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.755419 10330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:56.755519 10244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.755627 10244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:56.755817 10244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.755954 10244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:56.756599 10258 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:56.758525 10258 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260811 04:36:56.758639 10258 log.cc:1196] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260811 04:36:56.758903 10258 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Deleting consensus metadata
I20260811 04:36:56.759106 10258 master_runner.cc:315] Copying system tablet from 127.2.238.190:35291
I20260811 04:36:56.760314 10258 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: tablet copy: Beginning tablet copy session from remote peer at address 127.2.238.190:35291
I20260811 04:36:56.768501 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.760930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.2.238.190:35291)
0811 04:36:56.761055 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.761060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.761273 (+ 213us) server_negotiation.cc:407] Connection header received
0811 04:36:56.761437 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.761440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.761489 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.761577 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.762205 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.762741 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.763802 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.764027 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.764149 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.765279 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.765282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.765285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.765316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.766765 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.767214 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.767217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.767218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.767298 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.767580 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.767585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.767586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.767778 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.767888 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.768215 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.768329 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:36:56.768638 10455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.760882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:41744)
0811 04:36:56.761180 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.761192 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.761325 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.761609 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.761612 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.761897 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.762021 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.762062 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.762859 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.762863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.763676 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.763687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.764156 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.764159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.764263 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.764792 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.764814 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.765132 (+ 318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.766881 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.766886 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.766889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.767096 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.767391 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.767395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.767397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.767485 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.767878 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.767881 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.767988 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.768152 (+ 164us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.768316 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":93,"threads_started":1}
I20260811 04:36:56.769394 9670 tablet_copy_service.cc:140] P a21e91154bc8434ea619e020161ae1c5: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8f20a8835d7546818bb1c212a5abdcdb ({username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:41744)
I20260811 04:36:56.769502 9670 tablet_copy_service.cc:161] P a21e91154bc8434ea619e020161ae1c5: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8f20a8835d7546818bb1c212a5abdcdb at {username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:41744: session id = 8f20a8835d7546818bb1c212a5abdcdb-00000000000000000000000000000000
I20260811 04:36:56.770495 9670 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Tablet Copy: opened 0 blocks and 1 log segments
I20260811 04:36:56.771745 10258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260811 04:36:56.773670 10258 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: tablet copy: Starting download of 0 data blocks...
I20260811 04:36:56.773907 10258 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: tablet copy: Starting download of 1 WAL segments...
I20260811 04:36:56.775394 10258 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260811 04:36:56.777436 9670 tablet_copy_service.cc:342] P a21e91154bc8434ea619e020161ae1c5: Request end of tablet copy session 8f20a8835d7546818bb1c212a5abdcdb-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:41744
I20260811 04:36:56.777529 9670 tablet_copy_service.cc:433] P a21e91154bc8434ea619e020161ae1c5: ending tablet copy session 8f20a8835d7546818bb1c212a5abdcdb-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8f20a8835d7546818bb1c212a5abdcdb
I20260811 04:36:56.784163 10258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:56.785898 10459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.785939 10456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.786585 10258 server_base.cc:1034] running on GCE node
W20260811 04:36:56.786499 10457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.786867 10258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.787215 10258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.788353 10258 hybrid_clock.cc:648] HybridClock initialized: now 1786423016788346 us; error 25 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.793187 10258 init.cc:377] Logged in from keytab as oryx/127.2.238.188@KRBTEST.COM (short username oryx)
I20260811 04:36:56.793713 10258 webserver.cc:460] Webserver started at http://127.2.238.188:36901/ using document root <none> and password file <none>
I20260811 04:36:56.793949 10258 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.794010 10258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.795192 10258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.796780 10467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.797013 10258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:56.797132 10258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "8f20a8835d7546818bb1c212a5abdcdb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "9066c2986d1e53282ddb0f3088bcc839"
server_key_iv: "d4479c987fde78b9fc302fac05e61f9c"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.797281 10258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.823191 10258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.825565 10258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.830786 10258 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.188:35905
I20260811 04:36:56.830801 10522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.188:35905 every 8 connection(s)
I20260811 04:36:56.831414 10258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
I20260811 04:36:56.833009 10523 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:36:56.834034 10523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Bootstrap starting.
I20260811 04:36:56.842964 10523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:36:56.843276 10523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Bootstrap complete.
I20260811 04:36:56.843788 10523 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } }
I20260811 04:36:56.843925 10523 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8f20a8835d7546818bb1c212a5abdcdb, State: Initialized, Role: LEARNER
I20260811 04:36:56.844044 10523 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } }
I20260811 04:36:56.844391 10526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:56.844550 10523 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:36:56.844569 10526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: LEARNER
I20260811 04:36:56.846746 10537 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260811 04:36:56.871327 10441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:56.871659 10441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:56.871737 10441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 04:36:56.874704 9984 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
W20260811 04:36:56.877202 10441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:56.877262 10441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:56.877384 10441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:36:56.882757 10441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:40797
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=44691
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:56.883797 10441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:56.885059 10441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:56.895357 10542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.895361 10543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:56.895361 10546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:56.896067 10441 server_base.cc:1034] running on GCE node
I20260811 04:36:56.896507 10441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:56.897229 10441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:56.898413 10441 hybrid_clock.cc:648] HybridClock initialized: now 1786423016898382 us; error 54 us; skew 500 ppm
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:56.901093 10441 init.cc:377] Logged in from keytab as oryx/127.2.238.129@KRBTEST.COM (short username oryx)
I20260811 04:36:56.902082 10441 webserver.cc:460] Webserver started at http://127.2.238.129:44691/ using document root <none> and password file <none>
I20260811 04:36:56.902678 10441 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:56.902808 10441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:56.906713 10441 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:36:56.908687 10553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.909958 10441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:36:56.910099 10441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "b9aec8dc0349403ca768eb48393bea0b"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "a26c8d86d8cc789dc5beb31851b72d85"
server_key_iv: "3a7c0d1c5398a1f0ad5ba7e1c476dce5"
server_key_version: "encryptionkey@0"
I20260811 04:36:56.910650 10441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:56.921434 10441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:56.924232 10441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:56.924432 10441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:56.925024 10441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:56.925956 10441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:56.926023 10441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.926090 10441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:56.926141 10441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:56.936101 10441 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40797
I20260811 04:36:56.936125 10672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40797 every 8 connection(s)
I20260811 04:36:56.937165 10441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:36:56.938901 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 10441
I20260811 04:36:56.939100 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9851
I20260811 04:36:56.949291 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:44123
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:35291
--webserver_port=32833
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
Aug 11 04:36:56 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423016, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:36:56.958182 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.941462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:46841 (local address 127.2.238.188:35905)
0811 04:36:56.941758 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.941764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.942310 (+ 546us) server_negotiation.cc:407] Connection header received
0811 04:36:56.943401 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.943404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.943490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.943555 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.945724 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.946846 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.948625 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.948979 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949045 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.953573 (+ 4528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.953577 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.953579 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.953609 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.955933 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956480 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.956483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.956484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.956567 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956903 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.956906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.956908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.957134 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.957204 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.957795 (+ 591us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.957996 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
I20260811 04:36:56.958197 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.939275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:50801 (local address 127.2.238.189:43273)
0811 04:36:56.940389 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.940394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.941880 (+ 1486us) server_negotiation.cc:407] Connection header received
0811 04:36:56.943401 (+ 1521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.943404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.943490 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.943550 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.945861 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.946753 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.948603 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.948940 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949028 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.952888 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.952894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.952900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.952938 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.955200 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.955928 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.955935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.955939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.956047 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956544 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.956551 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.956555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.956856 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.956942 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.957848 (+ 906us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.957977 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015}
I20260811 04:36:56.958289 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.939454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:53035 (local address 127.2.238.190:35291)
0811 04:36:56.940704 (+ 1250us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:56.940709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:56.941951 (+ 1242us) server_negotiation.cc:407] Connection header received
0811 04:36:56.943413 (+ 1462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:56.943420 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:56.943521 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:56.943587 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:56.945733 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.946553 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.948626 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.948956 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949035 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.954285 (+ 5250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:56.954289 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:56.954291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:56.954325 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:56.956345 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956837 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.956843 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.956847 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.956928 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:56.957246 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:56.957251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:56.957254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:56.957515 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:56.957598 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:56.958019 (+ 421us) server_negotiation.cc:299] Negotiation successful
0811 04:36:56.958128 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1150}
I20260811 04:36:56.960160 10679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.939647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:53035)
0811 04:36:56.941847 (+ 2200us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.941865 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.943155 (+ 1290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.943975 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.943985 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.944658 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.945539 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.945562 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.946764 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.946767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.948486 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.948498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.949244 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.949572 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.950563 (+ 991us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.950588 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.954131 (+ 3543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.956468 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956471 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.956473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.956716 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.957045 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.957047 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.957049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.957123 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.957684 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.957689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.957795 (+ 106us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.958459 (+ 664us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.958824 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2048,"mutex_wait_us":1612,"spinlock_wait_cycles":72832,"thread_start_us":129,"threads_started":1}
I20260811 04:36:56.960187 10678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.941169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.2.238.129:46841)
0811 04:36:56.941933 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.941945 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.943166 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.943974 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.943983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.944658 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.945539 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.945562 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.946989 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.946992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.948486 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.948498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.949244 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949247 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.949576 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.950601 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.950612 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.953413 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.956079 (+ 2666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.956086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.956367 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.956701 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.956706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.956790 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.957283 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.957288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.957460 (+ 172us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.958459 (+ 999us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.959008 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":896,"spinlock_wait_cycles":40832,"thread_start_us":87,"threads_started":1}
I20260811 04:36:56.960719 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:56.939646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.2.238.129:50801)
0811 04:36:56.941695 (+ 2049us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:56.941753 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:56.943166 (+ 1413us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:56.943974 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:56.943983 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:56.944676 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:56.945538 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.945565 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.947031 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.947037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.948472 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:56.948486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:56.949245 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:56.949249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:56.949773 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:56.950748 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:56.950770 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:56.952609 (+ 1839us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:56.955353 (+ 2744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.955363 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.955382 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.955793 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.956194 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:56.956199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:56.956204 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:56.956413 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:56.957041 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:56.957051 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:56.957469 (+ 418us) client_negotiation.cc:769] Sending connection context
0811 04:36:56.958994 (+ 1525us) client_negotiation.cc:240] Negotiation successful
0811 04:36:56.959182 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1816,"mutex_wait_us":413,"spinlock_wait_cycles":93056,"thread_start_us":319,"threads_started":1}
I20260811 04:36:56.961619 10673 heartbeater.cc:347] Connected to a master server at 127.2.238.188:35905
I20260811 04:36:56.961664 10674 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:56.961839 10675 heartbeater.cc:347] Connected to a master server at 127.2.238.189:43273
I20260811 04:36:56.961930 10675 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:56.961930 10674 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:56.961941 10673 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:56.962574 10674 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:56.962576 10673 heartbeater.cc:511] Master 127.2.238.188:35905 requested a full tablet report, sending...
I20260811 04:36:56.962574 10675 heartbeater.cc:511] Master 127.2.238.189:43273 requested a full tablet report, sending...
I20260811 04:36:56.963944 9649 ts_manager.cc:194] Re-registered known tserver with Master: b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
I20260811 04:36:56.964298 10485 ts_manager.cc:194] Registered new tserver with Master: b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
I20260811 04:36:56.964387 10286 ts_manager.cc:194] Registered new tserver with Master: b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
I20260811 04:36:56.964710 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:53035
W20260811 04:36:57.068120 10681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:57.068421 10681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:57.068501 10681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:57.072736 10681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:57.072794 10681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:57.072850 10681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:36:57.079437 10681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:44123
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=32833
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:57.080930 10681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:57.082417 10681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:36:57.095177 10688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:57.095197 10691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:57.095371 10681 server_base.cc:1034] running on GCE node
W20260811 04:36:57.095180 10687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:57.096091 10681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:57.096944 10681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:57.098168 10681 hybrid_clock.cc:648] HybridClock initialized: now 1786423017098134 us; error 48 us; skew 500 ppm
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:57.101336 10681 init.cc:377] Logged in from keytab as oryx/127.2.238.130@KRBTEST.COM (short username oryx)
I20260811 04:36:57.102406 10681 webserver.cc:460] Webserver started at http://127.2.238.130:32833/ using document root <none> and password file <none>
I20260811 04:36:57.103225 10681 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:57.103340 10681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:57.103437 10132 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
I20260811 04:36:57.108759 10681 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:36:57.114154 10698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.115659 10681 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260811 04:36:57.115784 10681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
format_stamp: "Formatted at 2026-08-11 04:36:55 on dist-test-slave-0kls"
server_key: "4147a38de662745e68fa99e169c00045"
server_key_iv: "156ca51a0ef642938991fbe5d9d0cb9d"
server_key_version: "encryptionkey@0"
I20260811 04:36:57.116380 10681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:57.126729 10681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:57.129426 10681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:57.129608 10681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:57.130278 10681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:57.131145 10681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:57.131215 10681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.131278 10681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:57.131340 10681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.140455 10681 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:44123
I20260811 04:36:57.140480 10819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:44123 every 8 connection(s)
I20260811 04:36:57.141434 10681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:36:57.146996 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 10681
I20260811 04:36:57.147176 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9990
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
I20260811 04:36:57.158447 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.143299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:46761 (local address 127.2.238.190:35291)
0811 04:36:57.143463 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.143468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.144291 (+ 823us) server_negotiation.cc:407] Connection header received
0811 04:36:57.145486 (+ 1195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.145489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.145536 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.145624 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.147351 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.148041 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.149768 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.150317 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.150432 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.154550 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.154553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.154556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.154581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.156172 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.156711 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.156715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.156717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.156834 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.157245 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.157248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.157250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.157474 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.157588 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.158156 (+ 568us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.158283 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:36:57.158772 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.143343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:47977 (local address 127.2.238.189:43273)
0811 04:36:57.143461 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.143466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.144285 (+ 819us) server_negotiation.cc:407] Connection header received
0811 04:36:57.145383 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.145386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.145444 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.145540 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.149376 (+ 3836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.150154 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.151181 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.151446 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.151544 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.154947 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.154953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.154958 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.154989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.156724 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.157158 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.157162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.157163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.157253 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.157619 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.157625 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.157630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.157914 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.157995 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.158380 (+ 385us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.158514 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:36:57.159602 10825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.143643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.2.238.130:47977)
0811 04:36:57.144144 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.144191 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.145141 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.145707 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.145715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.146385 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.147177 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.147197 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.150302 (+ 3105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.150308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.151031 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.151039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.151669 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.151686 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.151804 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.152283 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.152303 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.154799 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.156835 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.156839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.156841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.157058 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.157373 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.157379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.157384 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.157485 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.158075 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.158081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.158190 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.158926 (+ 736us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.159092 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":120,"spinlock_wait_cycles":69120,"thread_start_us":164,"threads_started":1}
I20260811 04:36:57.159601 10824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.143643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:46761)
0811 04:36:57.144145 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.144191 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.145141 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.145708 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.145715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.146385 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.147175 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.147196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.148201 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.148206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.149569 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.149581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.150695 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.150703 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.151013 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.151711 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.151741 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.154352 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.156293 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.156301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.156316 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.156592 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.156926 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.156932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.156938 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.157147 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.157589 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.157597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.157917 (+ 320us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.158112 (+ 195us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.158445 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":65,"spinlock_wait_cycles":27904,"thread_start_us":150,"threads_started":1}
I20260811 04:36:57.160928 10821 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:57.161164 10821 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.161671 10821 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:57.161875 10822 heartbeater.cc:347] Connected to a master server at 127.2.238.189:43273
I20260811 04:36:57.161957 10822 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.162128 10822 heartbeater.cc:511] Master 127.2.238.189:43273 requested a full tablet report, sending...
I20260811 04:36:57.162878 9649 ts_manager.cc:194] Re-registered known tserver with Master: 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
I20260811 04:36:57.163120 10286 ts_manager.cc:194] Registered new tserver with Master: 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
I20260811 04:36:57.163619 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:46761
I20260811 04:36:57.163632 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:37629
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:35291
--webserver_port=34307
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
I20260811 04:36:57.171025 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.145031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:60021 (local address 127.2.238.188:35905)
0811 04:36:57.145166 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.145172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.145188 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:36:57.145664 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.145667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.145715 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.145840 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.163309 (+ 17469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.164020 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.164862 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.165273 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.165382 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.167691 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.167695 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.167698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.167729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.169209 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.169637 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.169641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.169642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.169729 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.170179 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.170183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.170185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.170374 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.170481 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.170748 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.170889 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 04:36:57.171375 10826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.144623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.2.238.130:60021)
0811 04:36:57.144868 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.144881 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.145184 (+ 303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.145851 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.145853 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.146385 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.147177 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.147196 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.164249 (+ 17053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.164251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.164742 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.164749 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.165544 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.165550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.165645 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.166212 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.166227 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.167547 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.169334 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.169337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.169339 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.169535 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.169982 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.169986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.169988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.170080 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.170484 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.170488 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.170583 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.170691 (+ 108us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.170842 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":172,"spinlock_wait_cycles":41600,"thread_start_us":90,"threads_started":1}
I20260811 04:36:57.172127 10820 heartbeater.cc:347] Connected to a master server at 127.2.238.188:35905
I20260811 04:36:57.172245 10820 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.172451 10820 heartbeater.cc:511] Master 127.2.238.188:35905 requested a full tablet report, sending...
I20260811 04:36:57.173198 10485 ts_manager.cc:194] Registered new tserver with Master: 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
W20260811 04:36:57.279317 10827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:36:57.279630 10827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:36:57.279726 10827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:36:57.283737 10827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:36:57.283824 10827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:36:57.283908 10827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:36:57.289042 10827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:37629
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=34307
--tserver_master_addrs=127.2.238.190:35291,127.2.238.189:43273,127.2.238.188:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:36:57.290273 10827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:36:57.291415 10827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:36:57.301815 10827 server_base.cc:1034] running on GCE node
W20260811 04:36:57.301720 10834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:57.301741 10833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:36:57.301720 10836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:36:57.302472 10827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:36:57.303169 10827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:36:57.304394 10827 hybrid_clock.cc:648] HybridClock initialized: now 1786423017304346 us; error 56 us; skew 500 ppm
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:36:57.307161 10827 init.cc:377] Logged in from keytab as oryx/127.2.238.131@KRBTEST.COM (short username oryx)
I20260811 04:36:57.308035 10827 webserver.cc:460] Webserver started at http://127.2.238.131:34307/ using document root <none> and password file <none>
I20260811 04:36:57.308636 10827 fs_manager.cc:359] Metadata directory not provided
I20260811 04:36:57.308755 10827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:36:57.312839 10827 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260811 04:36:57.314973 10844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.316210 10827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:36:57.316352 10827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "977af38b0ccd47429939f34995f7cdeb"
format_stamp: "Formatted at 2026-08-11 04:36:56 on dist-test-slave-0kls"
server_key: "62e426f2a863167867f424078946b9d2"
server_key_iv: "5a8a7f17de214706b22c2999af01f8e2"
server_key_version: "encryptionkey@0"
I20260811 04:36:57.316871 10827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:36:57.346024 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.338465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46508)
0811 04:36:57.338704 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.338717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.338823 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.339102 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.339104 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.339301 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.339401 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.339407 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.340396 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.340398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.340933 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.340940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.341470 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.341473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.341542 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.342099 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.342111 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.342451 (+ 340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.344371 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.344374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.344376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.344638 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.344962 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.344965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.344966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.345045 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.345454 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.345456 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.345540 (+ 84us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.345691 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.345883 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":81,"threads_started":1}
I20260811 04:36:57.346061 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.338531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46508 (local address 127.2.238.188:35905)
0811 04:36:57.338656 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.338661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.338791 (+ 130us) server_negotiation.cc:407] Connection header received
0811 04:36:57.338920 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.338922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.338976 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.339067 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.339501 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.340292 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.341036 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.341363 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.341452 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.342591 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.342595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.342597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.342623 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.344264 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.344726 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.344729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.344737 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.344849 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.345138 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.345141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.345142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.345348 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.345442 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.345770 (+ 328us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.345911 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:36:57.346313 10827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:36:57.347052 10496 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260811 04:36:57.347617 10850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21e91154bc8434ea619e020161ae1c5. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:57.347792 10850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: LEARNER
I20260811 04:36:57.350011 10827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:36:57.350207 10827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:36:57.350821 10827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:36:57.351725 10827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:36:57.351776 10827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.351842 10827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:36:57.351873 10827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:36:57.360903 10827 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:37629
I20260811 04:36:57.360924 10968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:37629 every 8 connection(s)
I20260811 04:36:57.362085 10827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:36:57.372036 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 10827
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.188@KRBTEST.COM
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
Aug 11 04:36:57 dist-test-slave-0kls krb5kdc[9601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423017, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.189@KRBTEST.COM
I20260811 04:36:57.379318 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.363951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:44857 (local address 127.2.238.190:35291)
0811 04:36:57.364127 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.364133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.365121 (+ 988us) server_negotiation.cc:407] Connection header received
0811 04:36:57.366302 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.366305 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.366365 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.366418 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.367849 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368631 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.369678 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370006 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370125 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.373927 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.373931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.373933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.373966 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.377359 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.377837 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.377841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.377842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.377931 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.378224 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.378228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.378230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.378426 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.378510 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.379022 (+ 512us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.379136 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:36:57.380417 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.364382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.2.238.131:41965)
0811 04:36:57.365344 (+ 962us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.365357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.366070 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.366621 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.366628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.367114 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.367665 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.367674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368524 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.368527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.369969 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.369981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370638 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370645 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.370743 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.371376 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.371394 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.374295 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.376636 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.376644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.376660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.377047 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.377416 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.377422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.377427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.377616 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.378157 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.378164 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.378531 (+ 367us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.378801 (+ 270us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.379206 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":319,"spinlock_wait_cycles":92672,"thread_start_us":175,"threads_started":1}
I20260811 04:36:57.381443 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.364375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:44857)
0811 04:36:57.364952 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.365004 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.366070 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.366621 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.366628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.367114 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.367667 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.367674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368764 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.368767 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.369551 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.369560 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370287 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.370503 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.371194 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.371210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.373726 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.377447 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.377450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.377452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.377698 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.378035 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.378038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.378040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.378124 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.378569 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.378572 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.378669 (+ 97us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.378799 (+ 130us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.380889 (+ 2090us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":1896,"spinlock_wait_cycles":34048,"thread_start_us":129,"threads_started":1}
I20260811 04:36:57.382591 10970 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35291
I20260811 04:36:57.382875 10970 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.382933 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.365475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:52489 (local address 127.2.238.188:35905)
0811 04:36:57.365593 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.365597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.365716 (+ 119us) server_negotiation.cc:407] Connection header received
0811 04:36:57.366297 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.366300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.366357 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.366454 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.367797 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368612 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370120 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370351 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370468 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.377968 (+ 7500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.377972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.377974 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.377997 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.380206 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.381260 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.381266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.381269 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.381402 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.381965 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.381969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.381970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.382221 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.382315 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.382656 (+ 341us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.382783 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:57.382968 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.363975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:41965 (local address 127.2.238.189:43273)
0811 04:36:57.364102 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.364107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.365435 (+ 1328us) server_negotiation.cc:407] Connection header received
0811 04:36:57.366383 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.366388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.366444 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.366510 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.367812 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368340 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370094 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370399 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370499 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.374455 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.374458 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.374462 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.374490 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.376506 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.377192 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.377196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.377199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.377300 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.377737 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.377742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.377745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.378009 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.378099 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.382696 (+ 4597us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.382823 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:57.383494 10970 heartbeater.cc:511] Master 127.2.238.190:35291 requested a full tablet report, sending...
I20260811 04:36:57.383550 10972 heartbeater.cc:347] Connected to a master server at 127.2.238.189:43273
I20260811 04:36:57.383668 10972 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.383833 10972 heartbeater.cc:511] Master 127.2.238.189:43273 requested a full tablet report, sending...
I20260811 04:36:57.384390 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.365364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.2.238.131:52489)
0811 04:36:57.365631 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.365643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.366076 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.366619 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.366625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.367105 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:36:57.367653 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.367667 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.368728 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.368731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.369971 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.369981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.370470 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.370474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.370570 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.371244 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:36:57.371265 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:36:57.377816 (+ 6551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:36:57.380357 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.380360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.380362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.381128 (+ 766us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.381728 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:36:57.381732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:36:57.381733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:36:57.381851 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:36:57.382373 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:36:57.382375 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:36:57.382462 (+ 87us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.383462 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.384208 (+ 746us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":173,"thread_start_us":84,"threads_started":1}
I20260811 04:36:57.384681 9649 ts_manager.cc:194] Re-registered known tserver with Master: 977af38b0ccd47429939f34995f7cdeb (127.2.238.131:37629)
I20260811 04:36:57.384768 10286 ts_manager.cc:194] Registered new tserver with Master: 977af38b0ccd47429939f34995f7cdeb (127.2.238.131:37629)
I20260811 04:36:57.384964 10969 heartbeater.cc:347] Connected to a master server at 127.2.238.188:35905
I20260811 04:36:57.385053 10969 heartbeater.cc:464] Registering TS with master...
I20260811 04:36:57.385166 10982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.374225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.2.238.190:35291)
0811 04:36:57.375289 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.375293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.375467 (+ 174us) server_negotiation.cc:407] Connection header received
0811 04:36:57.375630 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.375634 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.375687 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.375772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.376850 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.377660 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.378882 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.379197 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.379303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.381006 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.381011 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.381013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.381038 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.383035 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.383555 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.383559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.383560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.383662 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.383976 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.383979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.383981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.384199 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.384344 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.384880 (+ 536us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.385018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":66,"threads_started":1}
I20260811 04:36:57.385205 10969 heartbeater.cc:511] Master 127.2.238.188:35905 requested a full tablet report, sending...
I20260811 04:36:57.385397 9649 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:44857
I20260811 04:36:57.385985 10485 ts_manager.cc:194] Registered new tserver with Master: 977af38b0ccd47429939f34995f7cdeb (127.2.238.131:37629)
I20260811 04:36:57.391134 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.374427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.2.238.189:43273)
0811 04:36:57.375171 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.375176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.375773 (+ 597us) server_negotiation.cc:407] Connection header received
0811 04:36:57.375812 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.375815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.375859 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.375936 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.377212 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.380125 (+ 2913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.381315 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.381709 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.381865 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.387489 (+ 5624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.387492 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.387494 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.387517 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.389258 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.389734 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.389738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.389739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.389870 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.390163 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.390166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.390168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.390388 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.390509 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.390810 (+ 301us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.390940 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":651,"thread_start_us":75,"threads_started":1}
I20260811 04:36:57.388710 9649 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:57.393036 9649 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:57.395617 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.375032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46520 (local address 127.2.238.188:35905)
0811 04:36:57.376700 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.376704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.376717 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:57.376766 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.376771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.376824 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.376904 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.388002 (+ 11098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.388706 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.389631 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.390047 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.390162 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.392244 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.392248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.392251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.392284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.393926 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.394351 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.394355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.394356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.394444 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.394717 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.394720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.394722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.394922 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.395065 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.395291 (+ 226us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.395417 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":85,"threads_started":1}
I20260811 04:36:57.395735 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: NON_VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: true } } }
I20260811 04:36:57.395876 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:57.399648 10989 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: attempting to promote NON_VOTER 8f20a8835d7546818bb1c212a5abdcdb to VOTER
I20260811 04:36:57.400058 10989 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } }
I20260811 04:36:57.402487 10496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 LEARNER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 04:36:57.402729 10299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Refusing update from remote peer a21e91154bc8434ea619e020161ae1c5: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260811 04:36:57.402896 10446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 04:36:57.403177 10446 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260811 04:36:57.405055 10446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.405587 10496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.406991 11000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.407114 11000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:57.407114 10446 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.407277 10446 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:57.407371 11000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.407480 11000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:57.407896 10850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.408061 10850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:57.408344 10299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.410413 11001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a21e91154bc8434ea619e020161ae1c5" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:57.410614 11001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:57.419265 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.413665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:37629 (local address 127.0.0.1:60156)
0811 04:36:57.414439 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.414454 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.414545 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.415216 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.415221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.415243 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:57.415340 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.415346 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416643 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.416646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.417794 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.417838 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.418798 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.418801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.418887 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.418910 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.418973 (+ 63us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.419052 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":5,"thread_start_us":143,"threads_started":1}
I20260811 04:36:57.419273 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.414172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60156 (local address 127.2.238.131:37629)
0811 04:36:57.414521 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.414532 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.414553 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:36:57.414620 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.414624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.414808 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.414973 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:57.415440 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416530 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.417967 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.418646 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.418796 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.418971 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.419025 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.419118 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":122,"threads_started":1}
I20260811 04:36:57.419561 10849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.413158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40797 (local address 127.0.0.1:50674)
0811 04:36:57.413350 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.413367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.413449 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.415269 (+ 1820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.415272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.415319 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:57.415413 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.415419 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416559 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.416562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.418023 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.418032 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.419140 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.419144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.419241 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.419267 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.419322 (+ 55us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.419424 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":323}
I20260811 04:36:57.420166 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.413437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:44123 (local address 127.0.0.1:43820)
0811 04:36:57.414049 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.414064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.414164 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.414931 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.414934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.414954 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:57.415077 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.415083 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416514 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.416518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.418538 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.418550 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.419729 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.419733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.419854 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.419882 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.419946 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.420024 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":879,"thread_start_us":122,"threads_started":1}
I20260811 04:36:57.420218 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.413347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.2.238.129:40797)
0811 04:36:57.414795 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.414813 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.414836 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:36:57.414919 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.414924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.415118 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.415588 (+ 470us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:57.415703 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416388 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.418142 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.418997 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.419705 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.419910 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.419966 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.420071 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1301,"thread_start_us":110,"threads_started":1}
I20260811 04:36:57.420619 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.413918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.2.238.130:44123)
0811 04:36:57.414287 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.414300 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.414331 (+ 31us) server_negotiation.cc:407] Connection header received
0811 04:36:57.414494 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.414501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.414790 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.414974 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:57.415183 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.416367 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.418714 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.419588 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.419705 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.419989 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.420035 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.420115 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":144,"threads_started":1}
I20260811 04:36:57.423162 10899 tablet_service.cc:1467] Processing CreateTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 (DEFAULT_TABLE table=test-table [id=ee2bebc4be9a434eb4f543b9299b56d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:57.423162 10751 tablet_service.cc:1467] Processing CreateTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 (DEFAULT_TABLE table=test-table [id=ee2bebc4be9a434eb4f543b9299b56d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:57.424176 10751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:57.424177 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:57.428181 10604 tablet_service.cc:1467] Processing CreateTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 (DEFAULT_TABLE table=test-table [id=ee2bebc4be9a434eb4f543b9299b56d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:57.429327 10604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:57.431407 11007 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Bootstrap starting.
I20260811 04:36:57.434639 11007 tablet_bootstrap.cc:654] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:57.435387 11007 log.cc:824] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:57.436209 11008 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Bootstrap starting.
I20260811 04:36:57.436597 11010 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Bootstrap starting.
I20260811 04:36:57.437911 11007 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: No bootstrap required, opened a new log
I20260811 04:36:57.438122 11007 ts_tablet_manager.cc:1397] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:36:57.438740 11008 tablet_bootstrap.cc:654] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:57.439973 11008 log.cc:824] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:57.440428 11010 tablet_bootstrap.cc:654] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:57.441191 11010 log.cc:824] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:36:57.442435 11008 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: No bootstrap required, opened a new log
I20260811 04:36:57.442648 11008 ts_tablet_manager.cc:1397] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:36:57.443039 11010 tablet_bootstrap.cc:492] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: No bootstrap required, opened a new log
I20260811 04:36:57.443255 11010 ts_tablet_manager.cc:1397] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:36:57.443080 11007 raft_consensus.cc:348] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.443348 11007 raft_consensus.cc:374] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:57.443411 11007 raft_consensus.cc:729] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e81aeb392604767a3fe59a34b9cfbf3, State: Initialized, Role: FOLLOWER
I20260811 04:36:57.443902 11007 consensus_queue.cc:260] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.445156 11007 ts_tablet_manager.cc:1428] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260811 04:36:57.445660 10821 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
I20260811 04:36:57.447881 11010 raft_consensus.cc:348] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.448194 11010 raft_consensus.cc:374] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:57.448271 11010 raft_consensus.cc:729] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9aec8dc0349403ca768eb48393bea0b, State: Initialized, Role: FOLLOWER
I20260811 04:36:57.448763 11010 consensus_queue.cc:260] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.449915 11010 ts_tablet_manager.cc:1428] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260811 04:36:57.449882 11008 raft_consensus.cc:348] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.450174 11008 raft_consensus.cc:374] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:57.450232 11008 raft_consensus.cc:729] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977af38b0ccd47429939f34995f7cdeb, State: Initialized, Role: FOLLOWER
I20260811 04:36:57.450759 11008 consensus_queue.cc:260] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.451188 10674 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
I20260811 04:36:57.451798 11008 ts_tablet_manager.cc:1428] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260811 04:36:57.452450 10970 heartbeater.cc:503] Master 127.2.238.190:35291 was elected leader, sending a full tablet report...
W20260811 04:36:57.613615 10973 tablet.cc:2369] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:36:57.643071 10823 tablet.cc:2369] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:36:57.644888 11015 raft_consensus.cc:482] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:57.645180 11015 raft_consensus.cc:504] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.646754 11015 leader_election.cc:288] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797), 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
I20260811 04:36:57.650962 11005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.647460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:60845 (local address 127.2.238.130:44123)
0811 04:36:57.647774 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.647777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.647787 (+ 10us) server_negotiation.cc:407] Connection header received
0811 04:36:57.647829 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.647831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.647866 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.647952 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:57.648228 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.648780 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.649883 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.650297 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.650430 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.650490 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.650747 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.650825 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
I20260811 04:36:57.650962 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.647146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:37935 (local address 127.2.238.129:40797)
0811 04:36:57.647277 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.647280 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.647295 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:57.647530 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.647533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.647591 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.647681 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:57.647960 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.648544 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.649878 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.650299 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.650436 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.650496 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.650756 (+ 260us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.650833 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:36:57.650972 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.647397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:44123 (local address 127.2.238.131:60845)
0811 04:36:57.647523 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.647537 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.647634 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.647966 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.647969 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.647984 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:57.648104 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.648109 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.648891 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.648895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.649717 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.649727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.650423 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.650427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.650601 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.650630 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.650718 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.650826 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":109}
I20260811 04:36:57.650982 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.647072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40797 (local address 127.2.238.131:37935)
0811 04:36:57.647203 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:57.647221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:57.647351 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:57.647694 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:57.647697 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:57.647716 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:57.647849 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.647855 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.648652 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.648657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.649709 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:57.649719 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.650426 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.650429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:57.650629 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.650646 (+ 17us) client_negotiation.cc:769] Sending connection context
0811 04:36:57.650736 (+ 90us) client_negotiation.cc:240] Negotiation successful
0811 04:36:57.650833 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":52}
I20260811 04:36:57.651785 10626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de9ebc063f4cf4a176c7b3fbfd44e9" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9aec8dc0349403ca768eb48393bea0b" is_pre_election: true
I20260811 04:36:57.651849 10771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de9ebc063f4cf4a176c7b3fbfd44e9" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" is_pre_election: true
I20260811 04:36:57.652109 10626 raft_consensus.cc:2454] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 0.
I20260811 04:36:57.652139 10771 raft_consensus.cc:2454] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 0.
I20260811 04:36:57.652607 10846 leader_election.cc:302] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977af38b0ccd47429939f34995f7cdeb, b9aec8dc0349403ca768eb48393bea0b; no voters:
I20260811 04:36:57.652894 11015 raft_consensus.cc:2781] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:57.652987 11015 raft_consensus.cc:482] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:57.653041 11015 raft_consensus.cc:3037] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:57.654341 11015 raft_consensus.cc:504] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.654685 11015 leader_election.cc:288] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 election: Requested vote from peers b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797), 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123)
I20260811 04:36:57.655097 10626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de9ebc063f4cf4a176c7b3fbfd44e9" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9aec8dc0349403ca768eb48393bea0b"
I20260811 04:36:57.655097 10771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79de9ebc063f4cf4a176c7b3fbfd44e9" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
I20260811 04:36:57.655205 10626 raft_consensus.cc:3037] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:57.655233 10771 raft_consensus.cc:3037] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:57.656297 10626 raft_consensus.cc:2454] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 1.
I20260811 04:36:57.656548 10771 raft_consensus.cc:2454] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 1.
I20260811 04:36:57.656630 10846 leader_election.cc:302] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 977af38b0ccd47429939f34995f7cdeb, b9aec8dc0349403ca768eb48393bea0b; no voters:
I20260811 04:36:57.656855 11015 raft_consensus.cc:2781] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:57.657080 11015 raft_consensus.cc:686] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 1 LEADER]: Becoming Leader. State: Replica: 977af38b0ccd47429939f34995f7cdeb, State: Running, Role: LEADER
I20260811 04:36:57.657482 11015 consensus_queue.cc:237] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } }
I20260811 04:36:57.661238 9647 catalog_manager.cc:5495] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 977af38b0ccd47429939f34995f7cdeb (127.2.238.131). New cstate: current_term: 1 leader_uuid: "977af38b0ccd47429939f34995f7cdeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } health_report { overall_health: HEALTHY } } }
W20260811 04:36:57.689216 10676 tablet.cc:2369] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:36:57.710978 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.707785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60160 (local address 127.2.238.131:37629)
0811 04:36:57.707919 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.707923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.707936 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:57.708033 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.708035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.708093 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.708175 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:57.708393 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.708882 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.709639 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.709870 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.709954 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.710232 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:57.710657 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:57.710757 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.710813 (+ 56us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.710865 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:57.716843 10626 raft_consensus.cc:1261] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Refusing update from remote peer 977af38b0ccd47429939f34995f7cdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:57.716863 10771 raft_consensus.cc:1261] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Refusing update from remote peer 977af38b0ccd47429939f34995f7cdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:57.717574 11016 consensus_queue.cc:1035] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:57.717762 11015 consensus_queue.cc:1035] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:57.726842 11023 mvcc.cc:205] Tried to move back new op lower bound from 7317188680561041408 to 7317188680327888896. Current Snapshot: MvccSnapshot[applied={T|T < 7317188680561041408}]
I20260811 04:36:57.737447 9647 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:57.737660 9647 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:57.742218 9647 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a21e91154bc8434ea619e020161ae1c5: Sending DeleteTablet for 3 replicas of tablet 79de9ebc063f4cf4a176c7b3fbfd44e9
I20260811 04:36:57.743218 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:57 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:60156
I20260811 04:36:57.743424 10604 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:57 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:50674
I20260811 04:36:57.743553 10751 tablet_service.cc:1514] Processing DeleteTablet for tablet 79de9ebc063f4cf4a176c7b3fbfd44e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:57 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:43820
I20260811 04:36:57.743978 11032 tablet_replica.cc:330] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: stopping tablet replica
I20260811 04:36:57.744378 11032 raft_consensus.cc:2229] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:57.744686 11033 tablet_replica.cc:330] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: stopping tablet replica
I20260811 04:36:57.744818 11032 raft_consensus.cc:2258] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:57.745155 11033 raft_consensus.cc:2229] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:57.745664 11033 raft_consensus.cc:2258] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:57.746194 11031 tablet_replica.cc:330] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: stopping tablet replica
I20260811 04:36:57.746606 11031 raft_consensus.cc:2229] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:57.747026 11031 raft_consensus.cc:2258] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:57.747443 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 9615
I20260811 04:36:57.750749 11033 ts_tablet_manager.cc:1905] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:57.753731 11033 ts_tablet_manager.cc:1918] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:57.753868 11033 log.cc:1196] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/79de9ebc063f4cf4a176c7b3fbfd44e9
I20260811 04:36:57.754211 11033 ts_tablet_manager.cc:1939] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting consensus metadata
I20260811 04:36:57.755406 11032 ts_tablet_manager.cc:1905] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 04:36:57.758833 10700 connection.cc:535] server connection from 127.0.0.1:43820 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 04:36:57.760303 11032 ts_tablet_manager.cc:1918] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:57.760525 11032 log.cc:1196] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/79de9ebc063f4cf4a176c7b3fbfd44e9
I20260811 04:36:57.760556 11031 ts_tablet_manager.cc:1905] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:57.761013 11032 ts_tablet_manager.cc:1939] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P b9aec8dc0349403ca768eb48393bea0b: Deleting consensus metadata
W20260811 04:36:57.762343 10554 connection.cc:441] server connection from 127.0.0.1:50674 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50674 (request call id 1) could send its response
I20260811 04:36:57.763576 11031 ts_tablet_manager.cc:1918] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:57.763685 11031 log.cc:1196] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/79de9ebc063f4cf4a176c7b3fbfd44e9
I20260811 04:36:57.764008 11031 ts_tablet_manager.cc:1939] T 79de9ebc063f4cf4a176c7b3fbfd44e9 P 977af38b0ccd47429939f34995f7cdeb: Deleting consensus metadata
W20260811 04:36:57.765107 10847 connection.cc:441] server connection from 127.0.0.1:60156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60156 (request call id 1) could send its response
I20260811 04:36:57.768810 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.745991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55776 (local address 127.2.238.189:43273)
0811 04:36:57.746117 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.746122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.746136 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:36:57.746179 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.746181 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.746233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.746323 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.747456 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.748182 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.762349 (+ 14167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.762753 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.762862 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.764861 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.764865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.764867 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.764892 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.766924 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.767443 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.767446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.767448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.767542 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.767843 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.767847 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.767849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.768053 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.768149 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.768498 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.768630 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:36:57.776872 10418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.763011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.2.238.189:43273)
0811 04:36:57.763139 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.763143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.764860 (+ 1717us) server_negotiation.cc:407] Connection header received
0811 04:36:57.765000 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.765006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.765055 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.765111 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.766251 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.767093 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.768254 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.768559 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.768631 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.772747 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.772751 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.772753 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.772777 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.774858 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.775356 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.775361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.775366 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.775447 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.775806 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.775812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.775815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.776063 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.776169 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.776563 (+ 394us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.776694 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:36:57.777614 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.764113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.2.238.188:35905)
0811 04:36:57.764285 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.764290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.766080 (+ 1790us) server_negotiation.cc:407] Connection header received
0811 04:36:57.766258 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.766261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.766344 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.766396 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.767621 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.768369 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.770479 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.770836 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.770940 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.772743 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.772748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.772750 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.772778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.775026 (+ 2248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.775549 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.775554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.775555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.775658 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.775979 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.775982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.775984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.776222 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.776328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.776675 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.777463 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 04:36:57.779443 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:57.746281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.2.238.188:35905)
0811 04:36:57.746399 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:57.746404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:57.769405 (+ 23001us) server_negotiation.cc:407] Connection header received
0811 04:36:57.769552 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:57.769555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:57.769607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:57.769712 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:36:57.770728 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.771482 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.772575 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:57.772931 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:57.773463 (+ 532us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:57.775129 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:36:57.775133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:36:57.775137 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:36:57.775164 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:36:57.777008 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.778173 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.778177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.778179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.778274 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:36:57.778586 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:36:57.778589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:36:57.778591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:36:57.778795 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:36:57.778881 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:57.779165 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 04:36:57.779284 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:36:57.848464 10537 catalog_manager.cc:1259] Loaded cluster ID: 6fa29f39b6ff431882770b9e329403e9
I20260811 04:36:57.848541 10537 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: loading cluster ID for follower catalog manager: success
I20260811 04:36:57.850441 10537 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: acquiring CA information for follower catalog manager: success
I20260811 04:36:57.851986 10537 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:36:58.729586 11042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:58.729007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:43643)
0811 04:36:58.729341 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:58.729430 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":102,"threads_started":1}
W20260811 04:36:58.729890 10970 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:35291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:58.731750 11043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:58.730954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:45333)
0811 04:36:58.731266 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:58.731367 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":70,"threads_started":1}
I20260811 04:36:58.732126 11045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:58.731291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:35137)
0811 04:36:58.731934 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:58.732036 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":69,"threads_started":1}
W20260811 04:36:58.732528 10674 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:35291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
W20260811 04:36:58.732641 10821 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:35291 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:59.243546 11046 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a21e91154bc8434ea619e020161ae1c5)
I20260811 04:36:59.243669 11047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a21e91154bc8434ea619e020161ae1c5)
I20260811 04:36:59.243784 11046 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } }
I20260811 04:36:59.243952 11047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } }
I20260811 04:36:59.244798 11046 leader_election.cc:288] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291), 8f20a8835d7546818bb1c212a5abdcdb (127.2.238.188:35905)
I20260811 04:36:59.245000 11047 leader_election.cc:288] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291), 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189:43273)
I20260811 04:36:59.245450 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.244975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42242)
0811 04:36:59.245274 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.245345 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":72,"threads_started":1}
I20260811 04:36:59.245579 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.245080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42254)
0811 04:36:59.245407 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.245477 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
W20260811 04:36:59.245738 10471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
W20260811 04:36:59.245939 10275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:59.246807 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.246568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42264)
0811 04:36:59.246698 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.246747 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 04:36:59.247080 10471 leader_election.cc:334] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291): Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:59.247170 11051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.246951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42268)
0811 04:36:59.247059 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.247111 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:36:59.247282 10275 leader_election.cc:334] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291): Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:59.249660 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.245232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:43273 (local address 127.0.0.1:55786)
0811 04:36:59.245993 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.246008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:59.246106 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:59.246429 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:59.246432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:59.246450 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:59.246558 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.246563 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.247539 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.247542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.248373 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.248382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.249249 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.249252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.249339 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.249384 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 04:36:59.249465 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 04:36:59.249546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":44,"threads_started":1}
I20260811 04:36:59.249681 11052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.245473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55786 (local address 127.2.238.189:43273)
0811 04:36:59.246120 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.246125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.246141 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:36:59.246258 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.246266 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.246316 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.246416 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:59.246707 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.247428 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.248479 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.249138 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.249212 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.249260 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.249484 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.249575 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":454,"thread_start_us":67,"threads_started":1}
I20260811 04:36:59.250161 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f20a8835d7546818bb1c212a5abdcdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33d56bc0cbf94c25a30c5382b53e5360" is_pre_election: true
I20260811 04:36:59.250347 10299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f20a8835d7546818bb1c212a5abdcdb in term 1.
I20260811 04:36:59.250492 11048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.245328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35905 (local address 127.0.0.1:46548)
0811 04:36:59.245610 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.245625 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:59.245721 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:59.246530 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:59.246533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:59.246555 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:59.246695 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.246702 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.247906 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.247908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.248937 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.248947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.250017 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.250021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.250162 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.250191 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 04:36:59.250297 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 04:36:59.250368 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":38,"thread_start_us":50,"threads_started":1}
I20260811 04:36:59.250610 11053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.245674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46548 (local address 127.2.238.188:35905)
0811 04:36:59.246256 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.246260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.246273 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:36:59.246315 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.246318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.246361 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.246815 (+ 454us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:59.247002 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.247807 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.249045 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.249644 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.249905 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.250056 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.250337 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.250427 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":76,"threads_started":1}
I20260811 04:36:59.250741 10469 leader_election.cc:302] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33d56bc0cbf94c25a30c5382b53e5360, 8f20a8835d7546818bb1c212a5abdcdb; no voters: a21e91154bc8434ea619e020161ae1c5
I20260811 04:36:59.250963 11047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260811 04:36:59.250942 10496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "33d56bc0cbf94c25a30c5382b53e5360" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8f20a8835d7546818bb1c212a5abdcdb" is_pre_election: true
I20260811 04:36:59.251015 11047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Starting leader election (detected failure of leader a21e91154bc8434ea619e020161ae1c5)
I20260811 04:36:59.251053 11047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:36:59.252100 11047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } }
I20260811 04:36:59.252326 10496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 33d56bc0cbf94c25a30c5382b53e5360 in current term 2: Already voted for candidate 8f20a8835d7546818bb1c212a5abdcdb in this term.
I20260811 04:36:59.252491 11047 leader_election.cc:288] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 election: Requested vote from peers a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291), 33d56bc0cbf94c25a30c5382b53e5360 (127.2.238.189:43273)
I20260811 04:36:59.252856 10299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f20a8835d7546818bb1c212a5abdcdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "33d56bc0cbf94c25a30c5382b53e5360"
I20260811 04:36:59.252939 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.252709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42282)
0811 04:36:59.252856 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.252894 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260811 04:36:59.252977 10299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:36:59.253294 10275 leader_election.cc:302] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 33d56bc0cbf94c25a30c5382b53e5360; no voters: 8f20a8835d7546818bb1c212a5abdcdb, a21e91154bc8434ea619e020161ae1c5
I20260811 04:36:59.253726 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.253486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42294)
0811 04:36:59.253619 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.253666 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260811 04:36:59.253841 10471 leader_election.cc:334] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291): Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
I20260811 04:36:59.254163 10299 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f20a8835d7546818bb1c212a5abdcdb in term 2.
I20260811 04:36:59.254303 11046 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:36:59.254451 10469 leader_election.cc:302] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 33d56bc0cbf94c25a30c5382b53e5360, 8f20a8835d7546818bb1c212a5abdcdb; no voters: a21e91154bc8434ea619e020161ae1c5
I20260811 04:36:59.254614 11047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 2 FOLLOWER]: Leader election won for term 2
I20260811 04:36:59.254853 11047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [term 2 LEADER]: Becoming Leader. State: Replica: 8f20a8835d7546818bb1c212a5abdcdb, State: Running, Role: LEADER
I20260811 04:36:59.255139 11047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } }
I20260811 04:36:59.256011 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f20a8835d7546818bb1c212a5abdcdb. Latest consensus state: current_term: 2 leader_uuid: "8f20a8835d7546818bb1c212a5abdcdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:59.256116 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:59.256407 11058 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:36:59.258605 11058 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:36:59.258965 11058 catalog_manager.cc:1259] Loaded cluster ID: 6fa29f39b6ff431882770b9e329403e9
I20260811 04:36:59.259007 11058 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:36:59.259539 11058 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:36:59.259910 11058 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Loaded TSK: 0
I20260811 04:36:59.260583 11058 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:36:59.298300 10485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:36:59.301532 10485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:36:59.304823 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.304410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42310)
0811 04:36:59.304621 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.304714 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260811 04:36:59.304859 10299 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [term 2 FOLLOWER]: Refusing update from remote peer 8f20a8835d7546818bb1c212a5abdcdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260811 04:36:59.305413 11047 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260811 04:36:59.305696 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.305437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.0.0.1:42312)
0811 04:36:59.305563 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.305627 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 04:36:59.305907 10471 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb -> Peer a21e91154bc8434ea619e020161ae1c5 (127.2.238.190:35291): Couldn't send request to peer a21e91154bc8434ea619e020161ae1c5. Status: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260811 04:36:59.306981 11046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f20a8835d7546818bb1c212a5abdcdb. Latest consensus state: current_term: 2 leader_uuid: "8f20a8835d7546818bb1c212a5abdcdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:59.307106 11046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:59.308315 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "8f20a8835d7546818bb1c212a5abdcdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:59.308446 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb [sys.catalog]: This master's current role is: LEADER
I20260811 04:36:59.308682 11046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "8f20a8835d7546818bb1c212a5abdcdb" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a21e91154bc8434ea619e020161ae1c5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35291 } } peers { permanent_uuid: "33d56bc0cbf94c25a30c5382b53e5360" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 43273 } attrs { promote: false } } peers { permanent_uuid: "8f20a8835d7546818bb1c212a5abdcdb" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35905 } attrs { promote: false } } }
I20260811 04:36:59.308808 11060 mvcc.cc:205] Tried to move back new op lower bound from 7317188687067144192 to 7317188686871142400. Current Snapshot: MvccSnapshot[applied={T|T < 7317188687067144192}]
I20260811 04:36:59.308826 11046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d56bc0cbf94c25a30c5382b53e5360 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:36:59.319628 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.314667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:37629 (local address 127.0.0.1:60174)
0811 04:36:59.314863 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.314881 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:59.314978 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:59.315993 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:59.315996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:59.316021 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:59.316280 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.316287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.317187 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.317190 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.318280 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.318290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.319063 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.319066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.319217 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.319322 (+ 105us) client_negotiation.cc:769] Sending connection context
0811 04:36:59.319402 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 04:36:59.319482 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":194,"spinlock_wait_cycles":2176}
I20260811 04:36:59.320166 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.315268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40797 (local address 127.0.0.1:50680)
0811 04:36:59.315632 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.315646 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:59.315736 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:59.316522 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:59.316525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:59.316541 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:59.316628 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.316633 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.317534 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.317537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.318767 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.318789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.319690 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.319703 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.319820 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.319849 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 04:36:59.319911 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 04:36:59.319987 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260811 04:36:59.320151 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.315355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.2.238.129:40797)
0811 04:36:59.316206 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.316209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.316220 (+ 11us) server_negotiation.cc:407] Connection header received
0811 04:36:59.316267 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.316269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.316308 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.316412 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:59.316779 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.317407 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.318920 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.319565 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.319679 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.319887 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.319933 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.320008 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775,"thread_start_us":63,"threads_started":1}
I20260811 04:36:59.320637 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.314966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:44123 (local address 127.0.0.1:43832)
0811 04:36:59.315256 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.315270 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:36:59.315347 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:36:59.316000 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:36:59.316003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:36:59.316026 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:36:59.316280 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.316287 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.318051 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.318054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.319278 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:36:59.319290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.320204 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.320208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:36:59.320338 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.320379 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 04:36:59.320428 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 04:36:59.320504 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":135,"spinlock_wait_cycles":4480}
I20260811 04:36:59.320639 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.314808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.2.238.131:37629)
0811 04:36:59.315724 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.315729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.315744 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:36:59.315812 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.315815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.315864 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.315971 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:59.316410 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.317081 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.318420 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.318947 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.319056 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.319128 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.320098 (+ 970us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.320512 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":835,"thread_start_us":98,"threads_started":1}
I20260811 04:36:59.320860 10604 tablet_service.cc:1467] Processing CreateTablet for tablet 32a5a697ffca4747925d6d4416f53b6c (DEFAULT_TABLE table=test-table [id=f679b74fc768485db0dd19714288fccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:59.321151 10899 tablet_service.cc:1467] Processing CreateTablet for tablet 32a5a697ffca4747925d6d4416f53b6c (DEFAULT_TABLE table=test-table [id=f679b74fc768485db0dd19714288fccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:59.321223 10604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a5a697ffca4747925d6d4416f53b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:59.321347 11067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.315522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.2.238.130:44123)
0811 04:36:59.315765 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.315771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.315806 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:36:59.315851 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.315854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.315906 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.315990 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:36:59.317205 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.317912 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.319384 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.320064 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.321005 (+ 941us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.321070 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.321108 (+ 38us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.321186 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260811 04:36:59.321444 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a5a697ffca4747925d6d4416f53b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:59.321846 10751 tablet_service.cc:1467] Processing CreateTablet for tablet 32a5a697ffca4747925d6d4416f53b6c (DEFAULT_TABLE table=test-table [id=f679b74fc768485db0dd19714288fccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:36:59.322104 10751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a5a697ffca4747925d6d4416f53b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:36:59.324198 11069 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Bootstrap starting.
I20260811 04:36:59.324688 11070 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Bootstrap starting.
I20260811 04:36:59.324853 11068 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: Bootstrap starting.
I20260811 04:36:59.325631 11070 tablet_bootstrap.cc:654] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:59.325744 11068 tablet_bootstrap.cc:654] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:59.325958 11069 tablet_bootstrap.cc:654] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Neither blocks nor log segments found. Creating new log.
I20260811 04:36:59.327129 11068 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: No bootstrap required, opened a new log
I20260811 04:36:59.327232 11068 ts_tablet_manager.cc:1397] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 04:36:59.327267 11069 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: No bootstrap required, opened a new log
I20260811 04:36:59.327317 11070 tablet_bootstrap.cc:492] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: No bootstrap required, opened a new log
I20260811 04:36:59.327371 11069 ts_tablet_manager.cc:1397] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:59.327412 11070 ts_tablet_manager.cc:1397] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:36:59.327759 11069 raft_consensus.cc:348] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.327757 11068 raft_consensus.cc:348] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.327896 11069 raft_consensus.cc:374] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:59.327919 11068 raft_consensus.cc:374] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:59.327859 11070 raft_consensus.cc:348] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.327939 11069 raft_consensus.cc:729] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 977af38b0ccd47429939f34995f7cdeb, State: Initialized, Role: FOLLOWER
I20260811 04:36:59.327952 11068 raft_consensus.cc:729] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9aec8dc0349403ca768eb48393bea0b, State: Initialized, Role: FOLLOWER
I20260811 04:36:59.327967 11070 raft_consensus.cc:374] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:36:59.328001 11070 raft_consensus.cc:729] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e81aeb392604767a3fe59a34b9cfbf3, State: Initialized, Role: FOLLOWER
I20260811 04:36:59.328100 11068 consensus_queue.cc:260] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.328094 11069 consensus_queue.cc:260] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.328176 11070 consensus_queue.cc:260] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.328454 11068 ts_tablet_manager.cc:1428] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260811 04:36:59.328511 11070 ts_tablet_manager.cc:1428] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:36:59.330863 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.329771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:46501)
0811 04:36:59.330672 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.330735 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":93,"threads_started":1}
I20260811 04:36:59.331420 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.329074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:44355)
0811 04:36:59.331227 (+ 2153us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.331307 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2086,"thread_start_us":93,"threads_started":1}
I20260811 04:36:59.331719 10820 heartbeater.cc:503] Master 127.2.238.188:35905 was elected leader, sending a full tablet report...
I20260811 04:36:59.332892 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.332648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:41499)
0811 04:36:59.332796 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.332842 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260811 04:36:59.334090 11069 ts_tablet_manager.cc:1428] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260811 04:36:59.334333 10969 heartbeater.cc:503] Master 127.2.238.188:35905 was elected leader, sending a full tablet report...
I20260811 04:36:59.334580 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.329307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:54669)
0811 04:36:59.334421 (+ 5114us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.334484 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5037,"thread_start_us":107,"threads_started":1}
I20260811 04:36:59.334731 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.334506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:49907)
0811 04:36:59.334606 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.334666 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 04:36:59.335876 10673 heartbeater.cc:503] Master 127.2.238.188:35905 was elected leader, sending a full tablet report...
I20260811 04:36:59.336421 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.336141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:33185)
0811 04:36:59.336297 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.336351 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 04:36:59.350031 11075 raft_consensus.cc:482] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:36:59.350163 11075 raft_consensus.cc:504] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.350593 11075 leader_election.cc:288] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123), b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
I20260811 04:36:59.351083 10771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a5a697ffca4747925d6d4416f53b6c" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" is_pre_election: true
I20260811 04:36:59.351137 10626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a5a697ffca4747925d6d4416f53b6c" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9aec8dc0349403ca768eb48393bea0b" is_pre_election: true
I20260811 04:36:59.351222 10771 raft_consensus.cc:2454] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 0.
I20260811 04:36:59.351361 10626 raft_consensus.cc:2454] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 0.
I20260811 04:36:59.351619 10846 leader_election.cc:302] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e81aeb392604767a3fe59a34b9cfbf3, 977af38b0ccd47429939f34995f7cdeb; no voters:
I20260811 04:36:59.351796 11075 raft_consensus.cc:2781] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:36:59.351855 11075 raft_consensus.cc:482] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:36:59.351883 11075 raft_consensus.cc:3037] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:59.353287 11075 raft_consensus.cc:504] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.353642 11075 leader_election.cc:288] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 election: Requested vote from peers 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123), b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797)
I20260811 04:36:59.354094 10771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a5a697ffca4747925d6d4416f53b6c" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e81aeb392604767a3fe59a34b9cfbf3"
I20260811 04:36:59.354131 10626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a5a697ffca4747925d6d4416f53b6c" candidate_uuid: "977af38b0ccd47429939f34995f7cdeb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9aec8dc0349403ca768eb48393bea0b"
I20260811 04:36:59.354203 10771 raft_consensus.cc:3037] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:59.354290 10626 raft_consensus.cc:3037] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:36:59.355259 10626 raft_consensus.cc:2454] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 1.
I20260811 04:36:59.355295 10771 raft_consensus.cc:2454] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 977af38b0ccd47429939f34995f7cdeb in term 1.
I20260811 04:36:59.355607 10846 leader_election.cc:302] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e81aeb392604767a3fe59a34b9cfbf3, 977af38b0ccd47429939f34995f7cdeb; no voters:
I20260811 04:36:59.355829 11075 raft_consensus.cc:2781] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:36:59.356117 11075 raft_consensus.cc:686] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 1 LEADER]: Becoming Leader. State: Replica: 977af38b0ccd47429939f34995f7cdeb, State: Running, Role: LEADER
I20260811 04:36:59.356279 11075 consensus_queue.cc:237] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } }
I20260811 04:36:59.356949 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.356683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:43707)
0811 04:36:59.356811 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.356872 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260811 04:36:59.358111 10482 catalog_manager.cc:5495] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb reported cstate change: term changed from 0 to 1, leader changed from <none> to 977af38b0ccd47429939f34995f7cdeb (127.2.238.131). New cstate: current_term: 1 leader_uuid: "977af38b0ccd47429939f34995f7cdeb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "977af38b0ccd47429939f34995f7cdeb" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 } health_report { overall_health: UNKNOWN } } }
I20260811 04:36:59.371210 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.368112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.2.238.131:37629)
0811 04:36:59.368421 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0811 04:36:59.368424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:36:59.368434 (+ 10us) server_negotiation.cc:407] Connection header received
0811 04:36:59.368470 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:36:59.368472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:36:59.368510 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:36:59.368566 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:36:59.368851 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.369422 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.370304 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:36:59.370517 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:36:59.370594 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:36:59.370809 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:36:59.370875 (+ 66us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:36:59.370929 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 04:36:59.371005 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 04:36:59.371051 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236}
I20260811 04:36:59.373313 10626 raft_consensus.cc:1261] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Refusing update from remote peer 977af38b0ccd47429939f34995f7cdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:59.373313 10771 raft_consensus.cc:1261] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Refusing update from remote peer 977af38b0ccd47429939f34995f7cdeb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:36:59.373777 11075 consensus_queue.cc:1035] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9aec8dc0349403ca768eb48393bea0b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:59.374009 11080 consensus_queue.cc:1035] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e81aeb392604767a3fe59a34b9cfbf3" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 44123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:36:59.375823 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.375496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:44065)
0811 04:36:59.375681 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.375747 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260811 04:36:59.376184 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.375847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:41317)
0811 04:36:59.376054 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.376115 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260811 04:36:59.379998 11077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.379735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.131:44291)
0811 04:36:59.379870 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.379928 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260811 04:36:59.380962 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.380676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.130:44927)
0811 04:36:59.380840 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.380892 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260811 04:36:59.381774 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.381365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35291 (local address 127.2.238.129:45603)
0811 04:36:59.381642 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.381686 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:35291: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
I20260811 04:36:59.385902 10482 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:59.386059 10482 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:36:59.389423 10482 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f20a8835d7546818bb1c212a5abdcdb: Sending DeleteTablet for 3 replicas of tablet 32a5a697ffca4747925d6d4416f53b6c
I20260811 04:36:59.390192 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a5a697ffca4747925d6d4416f53b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:59 UTC) from {username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:60174
I20260811 04:36:59.390416 10751 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a5a697ffca4747925d6d4416f53b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:59 UTC) from {username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:43832
I20260811 04:36:59.390579 11096 tablet_replica.cc:330] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: stopping tablet replica
I20260811 04:36:59.390568 10604 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a5a697ffca4747925d6d4416f53b6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:36:59 UTC) from {username='oryx', principal='oryx/127.2.238.188@KRBTEST.COM'} at 127.0.0.1:50680
I20260811 04:36:59.390750 11096 raft_consensus.cc:2229] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:36:59.390870 11098 tablet_replica.cc:330] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: stopping tablet replica
I20260811 04:36:59.390969 11096 raft_consensus.cc:2258] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:59.390998 11098 raft_consensus.cc:2229] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:59.391259 11098 raft_consensus.cc:2258] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:36:59.391402 11099 tablet_replica.cc:330] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b: stopping tablet replica
I20260811 04:36:59.391532 11099 raft_consensus.cc:2229] T 32a5a697ffca4747925d6d4416f53b6c P b9aec8dc0349403ca768eb48393bea0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:36:59.391736 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 10441
I20260811 04:36:59.393621 11098 ts_tablet_manager.cc:1905] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:59.396250 11098 ts_tablet_manager.cc:1918] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:59.396363 11098 log.cc:1196] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/32a5a697ffca4747925d6d4416f53b6c
I20260811 04:36:59.396343 11096 ts_tablet_manager.cc:1905] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:36:59.396636 11098 ts_tablet_manager.cc:1939] T 32a5a697ffca4747925d6d4416f53b6c P 0e81aeb392604767a3fe59a34b9cfbf3: Deleting consensus metadata
I20260811 04:36:59.397351 10469 catalog_manager.cc:4841] TS 0e81aeb392604767a3fe59a34b9cfbf3 (127.2.238.130:44123): tablet 32a5a697ffca4747925d6d4416f53b6c (table test-table [id=f679b74fc768485db0dd19714288fccc]) successfully deleted
I20260811 04:36:59.399596 11096 ts_tablet_manager.cc:1918] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:36:59.399688 11096 log.cc:1196] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/32a5a697ffca4747925d6d4416f53b6c
I20260811 04:36:59.400063 11096 ts_tablet_manager.cc:1939] T 32a5a697ffca4747925d6d4416f53b6c P 977af38b0ccd47429939f34995f7cdeb: Deleting consensus metadata
I20260811 04:36:59.401830 10471 catalog_manager.cc:4841] TS 977af38b0ccd47429939f34995f7cdeb (127.2.238.131:37629): tablet 32a5a697ffca4747925d6d4416f53b6c (table test-table [id=f679b74fc768485db0dd19714288fccc]) successfully deleted
I20260811 04:36:59.404599 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 10681
I20260811 04:36:59.406741 11049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:36:59.404661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40797 (local address 127.0.0.1:50696)
0811 04:36:59.406615 (+ 1954us) negotiation.cc:107] Waiting for socket to connect
0811 04:36:59.406664 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40797: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1870}
W20260811 04:36:59.406950 10469 catalog_manager.cc:4584] TS b9aec8dc0349403ca768eb48393bea0b (127.2.238.129:40797): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 32a5a697ffca4747925d6d4416f53b6c: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40797: connect: Connection refused (error 111)
I20260811 04:36:59.415757 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 10827
I20260811 04:36:59.424260 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 10137
I20260811 04:36:59.431705 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 10258
2026-08-11T04:36:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5521 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:36:59 dist-test-slave-0kls krb5kdc[11103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:36:59 dist-test-slave-0kls krb5kdc[11103](info): set up 2 sockets
Aug 11 04:36:59 dist-test-slave-0kls krb5kdc[11103](info): commencing operation
krb5kdc: starting...
W20260811 04:37:00.510097 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:37:00 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:37:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab.
Entry for principal oryx/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab.
I20260811 04:37:00.644742 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:38285
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab
--principal=oryx/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:38285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:00.759117 11117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:00.759433 11117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:00.759511 11117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:00.763660 11117 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:00.763764 11117 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260811 04:37:00.763793 11117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:00.763837 11117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:00.763872 11117 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:00.763906 11117 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:00.769354 11117 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:38285
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.190.keytab
--principal=oryx/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:38285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:00.770637 11117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:00.772187 11117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:00.781189 11122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:00.781230 11123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:00.781189 11125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:00.781639 11117 server_base.cc:1034] running on GCE node
I20260811 04:37:00.782296 11117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:00.783486 11117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:00.784693 11117 hybrid_clock.cc:648] HybridClock initialized: now 1786423020784670 us; error 35 us; skew 500 ppm
Aug 11 04:37:00 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423020, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:00.787747 11117 init.cc:377] Logged in from keytab as oryx/127.2.238.190@KRBTEST.COM (short username oryx)
I20260811 04:37:00.788736 11117 webserver.cc:460] Webserver started at http://127.2.238.190:44863/ using document root <none> and password file <none>
I20260811 04:37:00.789361 11117 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:00.789459 11117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:00.789712 11117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:00.791944 11117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "1030e848120e43b9b4ae0f3a2c3ab996"
format_stamp: "Formatted at 2026-08-11 04:37:00 on dist-test-slave-0kls"
server_key: "ae1016d3b66493014c4afdcdf466b5a4"
server_key_iv: "d020bbb3ac99364b884409bc6b4315a2"
server_key_version: "encryptionkey@0"
I20260811 04:37:00.792546 11117 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "1030e848120e43b9b4ae0f3a2c3ab996"
format_stamp: "Formatted at 2026-08-11 04:37:00 on dist-test-slave-0kls"
server_key: "ae1016d3b66493014c4afdcdf466b5a4"
server_key_iv: "d020bbb3ac99364b884409bc6b4315a2"
server_key_version: "encryptionkey@0"
I20260811 04:37:00.796625 11117 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:37:00.799191 11132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:00.800483 11117 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:00.800632 11117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "1030e848120e43b9b4ae0f3a2c3ab996"
format_stamp: "Formatted at 2026-08-11 04:37:00 on dist-test-slave-0kls"
server_key: "ae1016d3b66493014c4afdcdf466b5a4"
server_key_iv: "d020bbb3ac99364b884409bc6b4315a2"
server_key_version: "encryptionkey@0"
I20260811 04:37:00.800747 11117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:00.820751 11117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:00.824154 11117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:00.824371 11117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:00.832770 11117 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:38285
I20260811 04:37:00.832803 11198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:38285 every 8 connection(s)
I20260811 04:37:00.833894 11117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:00.836716 11199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:00.842720 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11117
I20260811 04:37:00.842871 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:37:00.842952 11199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Bootstrap starting.
I20260811 04:37:00.843207 3002 external_mini_cluster.cc:1428] Setting key 843a3cf99c4eb92b6660d7e7de4c9f8e
I20260811 04:37:00.845551 11199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:00.846379 11199 log.cc:824] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:00.848496 11199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: No bootstrap required, opened a new log
Aug 11 04:37:00 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:00.853102 11199 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } }
I20260811 04:37:00.853375 11199 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:00.853440 11199 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1030e848120e43b9b4ae0f3a2c3ab996, State: Initialized, Role: FOLLOWER
I20260811 04:37:00.853969 11199 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } }
I20260811 04:37:00.854118 11199 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:00.854206 11199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:00.854339 11199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:00.855548 11199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } }
I20260811 04:37:00.855912 11199 leader_election.cc:302] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1030e848120e43b9b4ae0f3a2c3ab996; no voters:
I20260811 04:37:00.856315 11199 leader_election.cc:288] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:37:00.856451 11204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:00.856645 11204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [term 1 LEADER]: Becoming Leader. State: Replica: 1030e848120e43b9b4ae0f3a2c3ab996, State: Running, Role: LEADER
I20260811 04:37:00.856940 11204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } }
I20260811 04:37:00.857658 11199 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:00.858172 11205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1030e848120e43b9b4ae0f3a2c3ab996. Latest consensus state: current_term: 1 leader_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } } }
I20260811 04:37:00.858291 11205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:00.858719 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1030e848120e43b9b4ae0f3a2c3ab996" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 38285 } } }
I20260811 04:37:00.858831 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:00.859215 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:00.844623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46708 (local address 127.2.238.190:38285)
0811 04:37:00.845072 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:00.845100 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:00.845137 (+ 37us) server_negotiation.cc:407] Connection header received
0811 04:37:00.845908 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:00.845928 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:00.846267 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:00.846605 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:00.847479 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:00.848477 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:00.849623 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:00.850182 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:00.850301 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:00.853094 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:00.853099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:00.853113 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:00.853140 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:00.855324 (+ 2184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:00.855828 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:00.855833 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:00.855837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:00.855956 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:00.856303 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:00.856308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:00.856310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:00.856936 (+ 626us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:00.857131 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:00.857878 (+ 747us) server_negotiation.cc:299] Negotiation successful
0811 04:37:00.858225 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":155,"threads_started":1}
W20260811 04:37:00.862938 11219 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:37:00.863019 11219 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:37:00.863078 11209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:00.863961 11209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:00.870045 11209 catalog_manager.cc:1347] Generated new cluster ID: 3725c6a0f5c54205a1ce24e878fbc852
I20260811 04:37:00.870119 11209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:00.877291 11209 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:37:00.878265 11209 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:00.884382 11209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Generated new TSK 0
I20260811 04:37:00.885092 11209 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
I20260811 04:37:00.926695 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38285
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:01.065022 11226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:01.065307 11226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:01.065353 11226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:01.069480 11226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:01.069554 11226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:01.069602 11226 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:37:01.069708 11226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:01.075435 11226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:01.076776 11226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:01.077759 11226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:01.087885 11232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.087838 11233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.087838 11235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:01.088191 11226 server_base.cc:1034] running on GCE node
I20260811 04:37:01.088646 11226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:01.089324 11226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:01.090484 11226 hybrid_clock.cc:648] HybridClock initialized: now 1786423021090454 us; error 42 us; skew 500 ppm
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:01.098601 11226 init.cc:377] Logged in from keytab as oryx/127.2.238.129@KRBTEST.COM (short username oryx)
I20260811 04:37:01.099604 11226 webserver.cc:460] Webserver started at http://127.2.238.129:37785/ using document root <none> and password file <none>
I20260811 04:37:01.100292 11226 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:01.100394 11226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:01.100651 11226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:01.102934 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "ff5c293fc70b05869e57885c173de6a3"
server_key_iv: "376c448944ae094684b6121e27c02c12"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.103523 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "ff5c293fc70b05869e57885c173de6a3"
server_key_iv: "376c448944ae094684b6121e27c02c12"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.107805 11226 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:37:01.110347 11242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.111675 11226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:01.111840 11226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "ff5c293fc70b05869e57885c173de6a3"
server_key_iv: "376c448944ae094684b6121e27c02c12"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.111965 11226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:01.124641 11226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:01.127759 11226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:01.127986 11226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:01.129618 11226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:01.129698 11226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.129770 11226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:01.129848 11226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:01.141587 11226 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:45493
I20260811 04:37:01.141952 11364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:45493 every 8 connection(s)
I20260811 04:37:01.142809 11226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:01.144636 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.130588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:49721 (local address 127.2.238.190:38285)
0811 04:37:01.130786 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.130794 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.131718 (+ 924us) server_negotiation.cc:407] Connection header received
0811 04:37:01.132909 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.132914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.132980 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.133100 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.135075 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.135694 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.136928 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.137379 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.137522 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.140275 (+ 2753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.140278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.140281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.140320 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.142184 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.142780 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.142787 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.142790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.142894 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.143358 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.143361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.143363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.143665 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.143764 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.144353 (+ 589us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.144492 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 04:37:01.145134 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11226
I20260811 04:37:01.145288 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:37:01.145490 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.130997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.129:49721)
0811 04:37:01.131500 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.131552 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.132659 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.133389 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.133402 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.134009 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.134818 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.134847 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.135840 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.135844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.136758 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.136768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.137560 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.137564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.137757 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.138486 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.138503 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.140066 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.142329 (+ 2263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.142334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.142356 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.142656 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.143017 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.143020 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.143023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.143224 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.143819 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.143825 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.144104 (+ 279us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.144274 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.144535 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":147,"threads_started":1}
I20260811 04:37:01.145679 3002 external_mini_cluster.cc:1428] Setting key d5760315ed212facb47da2763d17cc89
I20260811 04:37:01.147315 11365 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38285
I20260811 04:37:01.147481 11365 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:01.148101 11365 heartbeater.cc:511] Master 127.2.238.190:38285 requested a full tablet report, sending...
I20260811 04:37:01.149977 11151 ts_manager.cc:194] Registered new tserver with Master: 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493)
I20260811 04:37:01.151425 11151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:49721
I20260811 04:37:01.158272 11372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.149323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.129:55803)
0811 04:37:01.149834 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.149852 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.150096 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.150458 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.150461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.150712 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.150851 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.150860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.151744 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.151747 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.152569 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.152580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.153112 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.153116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.153226 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.153902 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.153920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.154410 (+ 490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.156554 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.156556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.156558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.156763 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.157110 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.157112 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.157114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.157189 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.157692 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.157697 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.157831 (+ 134us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.157933 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.158046 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":131,"threads_started":1}
I20260811 04:37:01.158453 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.149914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:55803 (local address 127.2.238.190:38285)
0811 04:37:01.150041 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.150046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.150063 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:37:01.150260 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.150266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.150326 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.150415 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.150992 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.151611 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.152703 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.152965 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.153108 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.154558 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.154565 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.154571 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.154605 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.156439 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.156885 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.156890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.156894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.157000 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.157296 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.157300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.157302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.157568 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.157694 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.158077 (+ 383us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.158210 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":38}
WARNING: no policy specified for oryx/127.2.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
I20260811 04:37:01.178809 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38285
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:01.296757 11374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:01.297037 11374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:01.297083 11374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:01.301347 11374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:01.301424 11374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:01.301466 11374 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:37:01.301501 11374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:01.307380 11374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:01.308495 11374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:01.309604 11374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:01.320160 11380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.320190 11379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.320160 11382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:01.320515 11374 server_base.cc:1034] running on GCE node
I20260811 04:37:01.321054 11374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:01.321758 11374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:01.323010 11374 hybrid_clock.cc:648] HybridClock initialized: now 1786423021322988 us; error 31 us; skew 500 ppm
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:01.325845 11374 init.cc:377] Logged in from keytab as oryx/127.2.238.130@KRBTEST.COM (short username oryx)
I20260811 04:37:01.326836 11374 webserver.cc:460] Webserver started at http://127.2.238.130:36639/ using document root <none> and password file <none>
I20260811 04:37:01.327417 11374 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:01.327476 11374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:01.327651 11374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:01.329702 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "52fe18ced6234446a0dcce4b05ff4a73"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "9c39f385af783c985df4f5080234ef62"
server_key_iv: "820223022770e6a4c0dc96eff1525124"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.330322 11374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "52fe18ced6234446a0dcce4b05ff4a73"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "9c39f385af783c985df4f5080234ef62"
server_key_iv: "820223022770e6a4c0dc96eff1525124"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.334251 11374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:37:01.336923 11389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.338316 11374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:37:01.338445 11374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "52fe18ced6234446a0dcce4b05ff4a73"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "9c39f385af783c985df4f5080234ef62"
server_key_iv: "820223022770e6a4c0dc96eff1525124"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.338555 11374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:01.360523 11374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:01.363637 11374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:01.363837 11374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:01.365351 11374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:01.365403 11374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.365469 11374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:01.365512 11374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:01.376893 11374 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:38099
I20260811 04:37:01.376930 11516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:38099 every 8 connection(s)
I20260811 04:37:01.378180 11374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:01.380086 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.366289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:60521 (local address 127.2.238.190:38285)
0811 04:37:01.366433 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.366440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.367550 (+ 1110us) server_negotiation.cc:407] Connection header received
0811 04:37:01.368742 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.368748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.368836 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.368900 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.370651 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.371251 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.372446 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.372775 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.372901 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.376021 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.376025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.376028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.376066 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.377922 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.378525 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.378530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.378532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.378649 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.379065 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.379069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.379071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.379285 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.379367 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.379857 (+ 490us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.379964 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:37:01.380903 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.366881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.130:60521)
0811 04:37:01.367400 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.367444 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.368471 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.369394 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.369402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.369910 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.370494 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.370510 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.371389 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.371392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.372309 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.372320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.372967 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.372971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.373206 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.373936 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.373975 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.375836 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.378043 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.378049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.378065 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.378424 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.378786 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.378789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.378791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.378973 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.379384 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.379390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.379623 (+ 233us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.379776 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.380066 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":157,"threads_started":1}
I20260811 04:37:01.382156 11517 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38285
I20260811 04:37:01.382305 11517 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:01.382923 11517 heartbeater.cc:511] Master 127.2.238.190:38285 requested a full tablet report, sending...
I20260811 04:37:01.384183 11150 ts_manager.cc:194] Registered new tserver with Master: 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099)
I20260811 04:37:01.385041 11150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:60521
I20260811 04:37:01.386950 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11374
I20260811 04:37:01.387081 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:37:01.387377 3002 external_mini_cluster.cc:1428] Setting key b613d9af855216b277dedf22281ec548
I20260811 04:37:01.391940 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.384696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:54409 (local address 127.2.238.190:38285)
0811 04:37:01.384826 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.384830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.385001 (+ 171us) server_negotiation.cc:407] Connection header received
0811 04:37:01.385083 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.385086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.385137 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.385217 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.385845 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.386358 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.387260 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.387515 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.387642 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.388743 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.388746 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.388749 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.388772 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.390227 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.390675 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.390679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.390681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.390763 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.391052 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.391055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.391057 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.391292 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.391407 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.391679 (+ 272us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.391787 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:01.391963 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.384647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.130:54409)
0811 04:37:01.384922 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.384934 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.385018 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.385341 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.385343 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.385544 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.385667 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.385673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.386496 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.386500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.387081 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.387093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.387647 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.387650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.387742 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.388283 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.388294 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.388609 (+ 315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.390346 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.390350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.390352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.390556 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.390859 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.390864 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.390866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.390957 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.391395 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.391398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.391501 (+ 103us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.391622 (+ 121us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.391771 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
I20260811 04:37:01.414446 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:38285
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:01.525693 11526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:01.526021 11526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:01.526116 11526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:01.530083 11526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:01.530159 11526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:01.530227 11526 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260811 04:37:01.530309 11526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:01.535365 11526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.238.190:38285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:01.536424 11526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:01.537448 11526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:01.546999 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.546983 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:01.546983 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:01.547923 11526 server_base.cc:1034] running on GCE node
I20260811 04:37:01.548347 11526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:01.548956 11526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:01.550148 11526 hybrid_clock.cc:648] HybridClock initialized: now 1786423021550136 us; error 31 us; skew 500 ppm
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:01.552788 11526 init.cc:377] Logged in from keytab as oryx/127.2.238.131@KRBTEST.COM (short username oryx)
I20260811 04:37:01.553655 11526 webserver.cc:460] Webserver started at http://127.2.238.131:35965/ using document root <none> and password file <none>
I20260811 04:37:01.554279 11526 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:01.554369 11526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:01.554617 11526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:01.556689 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "359ce486ef6a41ba9eb23da67c839dc2"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "1ae3d468058a362038102ef9f67ce8f1"
server_key_iv: "78571a853c77573c4573c162170b3a91"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.557277 11526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "359ce486ef6a41ba9eb23da67c839dc2"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "1ae3d468058a362038102ef9f67ce8f1"
server_key_iv: "78571a853c77573c4573c162170b3a91"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.561210 11526 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:37:01.563544 11542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.564725 11526 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:01.564877 11526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "359ce486ef6a41ba9eb23da67c839dc2"
format_stamp: "Formatted at 2026-08-11 04:37:01 on dist-test-slave-0kls"
server_key: "1ae3d468058a362038102ef9f67ce8f1"
server_key_iv: "78571a853c77573c4573c162170b3a91"
server_key_version: "encryptionkey@0"
I20260811 04:37:01.564987 11526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:01.578034 11526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:01.580662 11526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:01.580854 11526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:01.582401 11526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:01.582469 11526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:01.582533 11526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:01.582567 11526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:01.593478 11526 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:40069
I20260811 04:37:01.593586 11662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:40069 every 8 connection(s)
I20260811 04:37:01.594461 11526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:01.596064 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.583033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53217 (local address 127.2.238.190:38285)
0811 04:37:01.583174 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.583179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.584122 (+ 943us) server_negotiation.cc:407] Connection header received
0811 04:37:01.585226 (+ 1104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.585230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.585326 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.585412 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.586998 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.587881 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.589228 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.589455 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.589569 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.592422 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.592425 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.592427 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.592450 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.594099 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.594681 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.594684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.594685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.594766 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.595159 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.595163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.595165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.595382 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.595460 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.595858 (+ 398us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.595951 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:37:01.596771 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.583512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.131:53217)
0811 04:37:01.583951 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.583998 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.584965 (+ 967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.585772 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.585824 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.586242 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.586846 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.586869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.588024 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.588029 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.589120 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.589132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.589620 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.589624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.589844 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.590429 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.590445 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.592247 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.594209 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.594214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.594228 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.594564 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.594897 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.594903 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.594905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.595065 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.595464 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.595468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.595665 (+ 197us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.595810 (+ 145us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.596019 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":128,"threads_started":1}
I20260811 04:37:01.597976 11663 heartbeater.cc:347] Connected to a master server at 127.2.238.190:38285
I20260811 04:37:01.598176 11663 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:01.598611 11663 heartbeater.cc:511] Master 127.2.238.190:38285 requested a full tablet report, sending...
I20260811 04:37:01.599553 11150 ts_manager.cc:194] Registered new tserver with Master: 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069)
I20260811 04:37:01.600260 11150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:53217
I20260811 04:37:01.600728 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11526
I20260811 04:37:01.600867 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:37:01.601348 3002 external_mini_cluster.cc:1428] Setting key 30c9fe422fa01c0a123a04d3dc56c2db
I20260811 04:37:01.603508 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:37:01.607704 11669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.600034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.2.238.131:55519)
0811 04:37:01.600332 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.600347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.600474 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.600701 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.600704 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.600925 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:01.601039 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.601045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.601903 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.601907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.602637 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.602647 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.603167 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.603169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.603268 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.603774 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:01.603786 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:01.604149 (+ 363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:01.606172 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.606177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.606182 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.606456 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.606730 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:01.606733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:01.606735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:01.606837 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:01.607201 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:01.607207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:01.607321 (+ 114us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.607428 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.607568 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":102,"threads_started":1}
I20260811 04:37:01.607999 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.600164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:55519 (local address 127.2.238.190:38285)
0811 04:37:01.600280 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.600283 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.600412 (+ 129us) server_negotiation.cc:407] Connection header received
0811 04:37:01.600565 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.600568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.600607 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.600685 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.601153 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.601731 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.602758 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.603059 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.603168 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.604289 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.604295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.604300 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.604326 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.606069 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.606565 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.606568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.606570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.606643 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.606930 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.606933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.606934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.607104 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.607177 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.607513 (+ 336us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.607635 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260811 04:37:01.629631 11202 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:01.623632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.2.238.190:38285)
0811 04:37:01.623761 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.623764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.623776 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:37:01.623813 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.623816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.623853 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.623915 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.624605 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.625145 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.626000 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.626206 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.626293 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.629405 (+ 3112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46710: BlockingRecv error: recv got EOF from 127.0.0.1:46710 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 04:37:01 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423021, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:01.638979 11202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.630978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46718 (local address 127.2.238.190:38285)
0811 04:37:01.631126 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.631131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.631267 (+ 136us) server_negotiation.cc:407] Connection header received
0811 04:37:01.631409 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.631412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.631455 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.631543 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:01.632132 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.632632 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.633562 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.633827 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.633922 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.636042 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:01.636046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:01.636048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:01.636080 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:01.637268 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.637676 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.637680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.637682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.637766 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:01.638118 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:01.638121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:01.638123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:01.638311 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:01.638444 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.638696 (+ 252us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.638824 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 04:37:01.641676 11150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:37:01.644824 11150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:01.659472 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.653154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44452 (local address 127.2.238.130:38099)
0811 04:37:01.653488 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.653500 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.653524 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:37:01.653671 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.653681 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.653925 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.654084 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:01.654988 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656207 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.658022 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.658666 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.658806 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.658982 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.659210 (+ 228us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.659320 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":112,"threads_started":1}
I20260811 04:37:01.659461 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.653991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40069 (local address 127.0.0.1:49392)
0811 04:37:01.654356 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.654368 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.654443 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.655274 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.655277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.655293 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:01.655368 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.655375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656322 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.656327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.657744 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.657757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.658912 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.658916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.659037 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.659066 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.659144 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.659334 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":251,"thread_start_us":104,"threads_started":1}
I20260811 04:37:01.659637 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.653659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.0.0.1:55364)
0811 04:37:01.653925 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.653939 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.654019 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.654749 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.654752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.654786 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:01.654910 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.654918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656343 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.656346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.657882 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.657887 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.659281 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.659284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.659367 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.659387 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.659439 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.659507 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":117,"thread_start_us":83,"threads_started":1}
I20260811 04:37:01.659703 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.654332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49392 (local address 127.2.238.131:40069)
0811 04:37:01.654864 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.654877 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.654899 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:37:01.654973 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.654980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.655139 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.655281 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:01.655490 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656213 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.657918 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.658751 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.659272 (+ 521us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.659431 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.659480 (+ 49us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.659558 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":132,"threads_started":1}
I20260811 04:37:01.660290 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.653880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.2.238.129:45493)
0811 04:37:01.654267 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.654273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.654296 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:37:01.654370 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.654375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.654581 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.654782 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:01.655034 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656207 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.658001 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.659114 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.659834 (+ 720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.659982 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.660036 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.660150 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":68,"threads_started":1}
I20260811 04:37:01.660989 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.652950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:38099 (local address 127.0.0.1:44452)
0811 04:37:01.653332 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.653386 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.653512 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.654673 (+ 1161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.654681 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.654709 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:01.654880 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.654888 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.656343 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.656346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.657873 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.657885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.658820 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.658824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.658927 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.659050 (+ 123us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.659177 (+ 127us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.660863 (+ 1686us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":145,"thread_start_us":117,"threads_started":1}
I20260811 04:37:01.662817 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec81ee7cb9451b94246f14106ce54c (DEFAULT_TABLE table=test-table [id=8cf7c5167a104a0b8162378710aa69f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.662810 11294 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec81ee7cb9451b94246f14106ce54c (DEFAULT_TABLE table=test-table [id=8cf7c5167a104a0b8162378710aa69f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.663161 11444 tablet_service.cc:1467] Processing CreateTablet for tablet 4bec81ee7cb9451b94246f14106ce54c (DEFAULT_TABLE table=test-table [id=8cf7c5167a104a0b8162378710aa69f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.663935 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec81ee7cb9451b94246f14106ce54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.663935 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec81ee7cb9451b94246f14106ce54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.664157 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bec81ee7cb9451b94246f14106ce54c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.670943 11691 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Bootstrap starting.
I20260811 04:37:01.671808 11692 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Bootstrap starting.
I20260811 04:37:01.672068 11693 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Bootstrap starting.
I20260811 04:37:01.673226 11691 tablet_bootstrap.cc:654] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.673936 11693 tablet_bootstrap.cc:654] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.673936 11692 tablet_bootstrap.cc:654] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.674073 11691 log.cc:824] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:01.674789 11693 log.cc:824] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:01.674789 11692 log.cc:824] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:01.676311 11691 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: No bootstrap required, opened a new log
I20260811 04:37:01.676510 11691 ts_tablet_manager.cc:1397] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:37:01.676996 11693 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: No bootstrap required, opened a new log
I20260811 04:37:01.677057 11692 tablet_bootstrap.cc:492] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: No bootstrap required, opened a new log
I20260811 04:37:01.677193 11693 ts_tablet_manager.cc:1397] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 04:37:01.677234 11692 ts_tablet_manager.cc:1397] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260811 04:37:01.680011 11693 raft_consensus.cc:348] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.680259 11693 raft_consensus.cc:374] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.680315 11693 raft_consensus.cc:729] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.680547 11691 raft_consensus.cc:348] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.680799 11691 raft_consensus.cc:374] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.680747 11693 consensus_queue.cc:260] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.680855 11691 raft_consensus.cc:729] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.681272 11691 consensus_queue.cc:260] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.681403 11692 raft_consensus.cc:348] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.681696 11692 raft_consensus.cc:374] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.681769 11692 raft_consensus.cc:729] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.681844 11693 ts_tablet_manager.cc:1428] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:37:01.682175 11663 heartbeater.cc:503] Master 127.2.238.190:38285 was elected leader, sending a full tablet report...
I20260811 04:37:01.682269 11691 ts_tablet_manager.cc:1428] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 04:37:01.682271 11692 consensus_queue.cc:260] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.682547 11517 heartbeater.cc:503] Master 127.2.238.190:38285 was elected leader, sending a full tablet report...
I20260811 04:37:01.683197 11692 ts_tablet_manager.cc:1428] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260811 04:37:01.683405 11365 heartbeater.cc:503] Master 127.2.238.190:38285 was elected leader, sending a full tablet report...
I20260811 04:37:01.764154 11697 raft_consensus.cc:482] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:01.764387 11697 raft_consensus.cc:504] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.765503 11697 leader_election.cc:288] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099), 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493)
I20260811 04:37:01.769550 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.765886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:38099 (local address 127.2.238.131:36401)
0811 04:37:01.766051 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.766066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.766203 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.766565 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.766568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.766587 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:01.766700 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.766705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.767459 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.767465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.768319 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.768329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.769092 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.769095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.769244 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.769262 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.769364 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.769437 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":24}
I20260811 04:37:01.769605 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.766042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:36401 (local address 127.2.238.130:38099)
0811 04:37:01.766191 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.766196 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.766213 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:37:01.766356 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.766363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.766424 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.766508 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:01.766822 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.767342 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.768447 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.768969 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.769097 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.769156 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.769382 (+ 226us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.769462 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260811 04:37:01.770328 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec81ee7cb9451b94246f14106ce54c" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73" is_pre_election: true
I20260811 04:37:01.770474 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.766209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.2.238.131:55599)
0811 04:37:01.766605 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:01.766618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:01.766743 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:01.766996 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:01.766998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:01.767014 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:01.767152 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.767158 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.767869 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.767875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.769182 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:01.769192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.770042 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.770045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:01.770145 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.770171 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:37:01.770268 (+ 97us) client_negotiation.cc:240] Negotiation successful
0811 04:37:01.770334 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":1,"thread_start_us":161,"threads_started":1}
I20260811 04:37:01.770474 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.766388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:55599 (local address 127.2.238.129:45493)
0811 04:37:01.766543 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.766547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.766714 (+ 167us) server_negotiation.cc:407] Connection header received
0811 04:37:01.766847 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.766850 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.766901 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.766984 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:01.767274 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.767760 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.769331 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.769918 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.770042 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.770089 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.770299 (+ 210us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.770369 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 04:37:01.770686 11465 raft_consensus.cc:2454] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:01.771204 11544 leader_election.cc:302] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 52fe18ced6234446a0dcce4b05ff4a73; no voters:
I20260811 04:37:01.771421 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec81ee7cb9451b94246f14106ce54c" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" is_pre_election: true
I20260811 04:37:01.771481 11697 raft_consensus.cc:2781] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:01.771580 11697 raft_consensus.cc:482] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:01.771636 11697 raft_consensus.cc:3037] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:01.771713 11315 raft_consensus.cc:2454] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:01.772897 11697 raft_consensus.cc:504] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.773291 11697 leader_election.cc:288] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099), 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493)
I20260811 04:37:01.773625 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec81ee7cb9451b94246f14106ce54c" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73"
I20260811 04:37:01.773689 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bec81ee7cb9451b94246f14106ce54c" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
I20260811 04:37:01.773769 11465 raft_consensus.cc:3037] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:01.773819 11315 raft_consensus.cc:3037] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:01.775148 11465 raft_consensus.cc:2454] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:01.775148 11315 raft_consensus.cc:2454] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:01.775488 11544 leader_election.cc:302] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 52fe18ced6234446a0dcce4b05ff4a73; no voters:
I20260811 04:37:01.775655 11697 raft_consensus.cc:2781] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:01.775931 11697 raft_consensus.cc:686] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Running, Role: LEADER
I20260811 04:37:01.776316 11697 consensus_queue.cc:237] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.779903 11151 catalog_manager.cc:5495] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "359ce486ef6a41ba9eb23da67c839dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } health_report { overall_health: HEALTHY } } }
I20260811 04:37:01.815274 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:01.811588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.2.238.131:40069)
0811 04:37:01.811734 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:01.811739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:01.811759 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:37:01.811849 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:01.811852 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:01.811904 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:01.812026 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:01.812300 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.812794 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.813700 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:01.813988 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:01.814111 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:01.814413 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:01.814852 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:01.814983 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:01.815054 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 04:37:01.815122 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:37:01.821046 11315 raft_consensus.cc:1261] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:01.821302 11465 raft_consensus.cc:1261] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:01.821777 11697 consensus_queue.cc:1035] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:01.822003 11702 consensus_queue.cc:1035] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:01.829010 11705 mvcc.cc:205] Tried to move back new op lower bound from 7317188697371525120 to 7317188697199087616. Current Snapshot: MvccSnapshot[applied={T|T < 7317188697371525120}]
I20260811 04:37:01.839660 11151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:01.839808 11151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:01.842337 11151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Sending DeleteTablet for 3 replicas of tablet 4bec81ee7cb9451b94246f14106ce54c
I20260811 04:37:01.843152 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec81ee7cb9451b94246f14106ce54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:01 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49392
I20260811 04:37:01.843228 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec81ee7cb9451b94246f14106ce54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:01 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55364
I20260811 04:37:01.843685 11716 tablet_replica.cc:330] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: stopping tablet replica
I20260811 04:37:01.843829 11444 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bec81ee7cb9451b94246f14106ce54c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:01 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:44452
I20260811 04:37:01.843968 11716 raft_consensus.cc:2229] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:01.844234 11716 raft_consensus.cc:2258] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:01.845094 11151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:37:01.845486 11151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:01.846192 11716 ts_tablet_manager.cc:1905] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:01.847777 11715 tablet_replica.cc:330] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: stopping tablet replica
I20260811 04:37:01.848048 11715 raft_consensus.cc:2229] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:37:01.848390 11715 raft_consensus.cc:2258] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:01.848994 11716 ts_tablet_manager.cc:1918] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:01.849109 11716 log.cc:1196] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/4bec81ee7cb9451b94246f14106ce54c
I20260811 04:37:01.849444 11716 ts_tablet_manager.cc:1939] T 4bec81ee7cb9451b94246f14106ce54c P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting consensus metadata
I20260811 04:37:01.849751 11715 ts_tablet_manager.cc:1905] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:01.850658 11134 catalog_manager.cc:4841] TS 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493): tablet 4bec81ee7cb9451b94246f14106ce54c (table test-table [id=8cf7c5167a104a0b8162378710aa69f5]) successfully deleted
I20260811 04:37:01.851316 11444 tablet_service.cc:1467] Processing CreateTablet for tablet c7ab70242a81412cb1d5709881a792df (DEFAULT_TABLE table=test-table [id=fd5061b799ae431b9aa40e63497e37cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.851359 11593 tablet_service.cc:1467] Processing CreateTablet for tablet c7ab70242a81412cb1d5709881a792df (DEFAULT_TABLE table=test-table [id=fd5061b799ae431b9aa40e63497e37cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.851604 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ab70242a81412cb1d5709881a792df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.851665 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ab70242a81412cb1d5709881a792df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.852366 11715 ts_tablet_manager.cc:1918] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:01.852451 11715 log.cc:1196] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/4bec81ee7cb9451b94246f14106ce54c
I20260811 04:37:01.852773 11715 ts_tablet_manager.cc:1939] T 4bec81ee7cb9451b94246f14106ce54c P 359ce486ef6a41ba9eb23da67c839dc2: Deleting consensus metadata
I20260811 04:37:01.853755 11693 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Bootstrap starting.
I20260811 04:37:01.854223 11691 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Bootstrap starting.
I20260811 04:37:01.854705 11693 tablet_bootstrap.cc:654] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.855146 11691 tablet_bootstrap.cc:654] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.855396 11134 catalog_manager.cc:4841] TS 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069): tablet 4bec81ee7cb9451b94246f14106ce54c (table test-table [id=8cf7c5167a104a0b8162378710aa69f5]) successfully deleted
I20260811 04:37:01.855875 11294 tablet_service.cc:1467] Processing CreateTablet for tablet c7ab70242a81412cb1d5709881a792df (DEFAULT_TABLE table=test-table [id=fd5061b799ae431b9aa40e63497e37cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:01.856060 11693 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: No bootstrap required, opened a new log
I20260811 04:37:01.856132 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ab70242a81412cb1d5709881a792df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:01.856169 11693 ts_tablet_manager.cc:1397] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:01.856616 11693 raft_consensus.cc:348] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.856719 11693 raft_consensus.cc:374] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.856748 11693 raft_consensus.cc:729] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.856851 11693 consensus_queue.cc:260] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.856945 11691 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: No bootstrap required, opened a new log
I20260811 04:37:01.857018 11691 ts_tablet_manager.cc:1397] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:37:01.857123 11693 ts_tablet_manager.cc:1428] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:01.857460 11691 raft_consensus.cc:348] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.857556 11691 raft_consensus.cc:374] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.857585 11691 raft_consensus.cc:729] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.857697 11691 consensus_queue.cc:260] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.858011 11691 ts_tablet_manager.cc:1428] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:01.858861 11692 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Bootstrap starting.
I20260811 04:37:01.859859 11717 tablet_replica.cc:330] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: stopping tablet replica
I20260811 04:37:01.859921 11692 tablet_bootstrap.cc:654] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:01.860343 11717 raft_consensus.cc:2229] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:01.860707 11717 raft_consensus.cc:2258] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:01.861207 11692 tablet_bootstrap.cc:492] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: No bootstrap required, opened a new log
I20260811 04:37:01.861315 11692 ts_tablet_manager.cc:1397] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260811 04:37:01.861709 11692 raft_consensus.cc:348] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.861841 11692 raft_consensus.cc:374] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:01.861886 11692 raft_consensus.cc:729] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Initialized, Role: FOLLOWER
I20260811 04:37:01.862035 11692 consensus_queue.cc:260] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:01.862315 11692 ts_tablet_manager.cc:1428] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:01.862238 11717 ts_tablet_manager.cc:1905] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:01.865500 11717 ts_tablet_manager.cc:1918] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:01.865610 11717 log.cc:1196] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/4bec81ee7cb9451b94246f14106ce54c
I20260811 04:37:01.865953 11717 ts_tablet_manager.cc:1939] T 4bec81ee7cb9451b94246f14106ce54c P 52fe18ced6234446a0dcce4b05ff4a73: Deleting consensus metadata
I20260811 04:37:01.867172 11134 catalog_manager.cc:4841] TS 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099): tablet 4bec81ee7cb9451b94246f14106ce54c (table test-table [id=8cf7c5167a104a0b8162378710aa69f5]) successfully deleted
W20260811 04:37:01.879408 11518 tablet.cc:2369] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:37:01.894657 11366 tablet.cc:2369] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:37:02.096216 11664 tablet.cc:2369] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:37:02.119148 11698 raft_consensus.cc:482] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:02.119333 11698 raft_consensus.cc:504] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.120733 11698 leader_election.cc:288] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069)
I20260811 04:37:02.125953 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.121464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:46455 (local address 127.2.238.131:40069)
0811 04:37:02.121598 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.121602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.121775 (+ 173us) server_negotiation.cc:407] Connection header received
0811 04:37:02.121863 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.121865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.121918 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.121977 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:02.122243 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.122949 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.124569 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.125209 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.125348 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.125408 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.125717 (+ 309us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.125831 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:37:02.125953 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.121038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:44961 (local address 127.2.238.129:45493)
0811 04:37:02.121194 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.121199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.121316 (+ 117us) server_negotiation.cc:407] Connection header received
0811 04:37:02.121530 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.121533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.121609 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.121679 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:02.122081 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.122876 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.124569 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.125208 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.125348 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.125408 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.125718 (+ 310us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.125819 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:37:02.125957 11723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.121348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40069 (local address 127.2.238.130:46455)
0811 04:37:02.121638 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.121652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.121763 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.122019 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.122024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.122047 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:02.122142 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.122146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.123059 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.123062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.124407 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.124418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.125344 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.125348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.125552 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.125607 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.125680 (+ 73us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.125762 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":167,"thread_start_us":82,"threads_started":1}
I20260811 04:37:02.125967 11722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.120903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.2.238.130:44961)
0811 04:37:02.121205 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.121222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.121392 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.121722 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.121725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.121773 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:02.121936 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.121942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.123001 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.123003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.124407 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.124418 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.125344 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.125348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.125547 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.125602 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.125680 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.125762 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":151,"spinlock_wait_cycles":768,"thread_start_us":88,"threads_started":1}
I20260811 04:37:02.126673 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ab70242a81412cb1d5709881a792df" candidate_uuid: "52fe18ced6234446a0dcce4b05ff4a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" is_pre_election: true
I20260811 04:37:02.126708 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ab70242a81412cb1d5709881a792df" candidate_uuid: "52fe18ced6234446a0dcce4b05ff4a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359ce486ef6a41ba9eb23da67c839dc2" is_pre_election: true
I20260811 04:37:02.126855 11315 raft_consensus.cc:2454] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52fe18ced6234446a0dcce4b05ff4a73 in term 0.
I20260811 04:37:02.126935 11614 raft_consensus.cc:2454] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 52fe18ced6234446a0dcce4b05ff4a73 in term 0.
I20260811 04:37:02.127324 11392 leader_election.cc:302] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fe18ced6234446a0dcce4b05ff4a73, 8bbad433f3cf44adad9a5fa0e46398cd; no voters:
I20260811 04:37:02.127597 11698 raft_consensus.cc:2781] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:02.127679 11698 raft_consensus.cc:482] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:02.127722 11698 raft_consensus.cc:3037] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.128808 11698 raft_consensus.cc:504] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.129194 11698 leader_election.cc:288] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [CANDIDATE]: Term 1 election: Requested vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069)
I20260811 04:37:02.129649 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ab70242a81412cb1d5709881a792df" candidate_uuid: "52fe18ced6234446a0dcce4b05ff4a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359ce486ef6a41ba9eb23da67c839dc2"
I20260811 04:37:02.129649 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ab70242a81412cb1d5709881a792df" candidate_uuid: "52fe18ced6234446a0dcce4b05ff4a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
I20260811 04:37:02.129810 11315 raft_consensus.cc:3037] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.129810 11614 raft_consensus.cc:3037] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.131089 11315 raft_consensus.cc:2454] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52fe18ced6234446a0dcce4b05ff4a73 in term 1.
I20260811 04:37:02.131166 11614 raft_consensus.cc:2454] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 52fe18ced6234446a0dcce4b05ff4a73 in term 1.
I20260811 04:37:02.131423 11392 leader_election.cc:302] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 52fe18ced6234446a0dcce4b05ff4a73; no voters:
I20260811 04:37:02.131671 11698 raft_consensus.cc:2781] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:02.131898 11698 raft_consensus.cc:686] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 LEADER]: Becoming Leader. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Running, Role: LEADER
I20260811 04:37:02.132182 11698 consensus_queue.cc:237] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.134663 11149 catalog_manager.cc:5495] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "52fe18ced6234446a0dcce4b05ff4a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } health_report { overall_health: UNKNOWN } } }
Aug 11 04:37:02 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423020, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.190@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:02.188036 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.179546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:38285 (local address 127.0.0.1:46724)
0811 04:37:02.179809 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.179823 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.179941 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.180199 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.180201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.180592 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:02.180693 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.180700 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.181427 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.181429 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.182096 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.182105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.182536 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.182540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.182614 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.183264 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:02.183275 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:02.184680 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:02.186468 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:02.186477 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:02.186483 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:02.186728 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:02.187037 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:02.187039 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:02.187041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:02.187117 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:02.187502 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:02.187508 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:02.187618 (+ 110us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.187710 (+ 92us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.187851 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":62,"threads_started":1}
I20260811 04:37:02.188053 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.179655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46724 (local address 127.2.238.190:38285)
0811 04:37:02.179911 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.179916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.179930 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:02.180034 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.180037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.180079 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.180156 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:02.180801 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.181329 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.182194 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.182410 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.182489 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.184813 (+ 2324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:02.184816 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:02.184818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:02.184839 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:02.186345 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:02.186856 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:02.186859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:02.186861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:02.186946 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:02.187208 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:02.187211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:02.187212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:02.187409 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:02.187487 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.187768 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.187916 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":188,"thread_start_us":63,"threads_started":1}
I20260811 04:37:02.192085 11149 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:46724:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260811 04:37:02.192745 11149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:02.197729 11444 tablet_service.cc:1467] Processing CreateTablet for tablet f911d1bf033f4e65a959dd11be28396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adc9bea72c240838fe9a78aedd99b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:37:02.197744 11294 tablet_service.cc:1467] Processing CreateTablet for tablet f911d1bf033f4e65a959dd11be28396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adc9bea72c240838fe9a78aedd99b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:37:02.197871 11593 tablet_service.cc:1467] Processing CreateTablet for tablet f911d1bf033f4e65a959dd11be28396e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8adc9bea72c240838fe9a78aedd99b31]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260811 04:37:02.198220 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911d1bf033f4e65a959dd11be28396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.198220 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911d1bf033f4e65a959dd11be28396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.198246 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f911d1bf033f4e65a959dd11be28396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.201030 11691 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: Bootstrap starting.
I20260811 04:37:02.201026 11693 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: Bootstrap starting.
I20260811 04:37:02.201099 11692 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: Bootstrap starting.
I20260811 04:37:02.201932 11692 tablet_bootstrap.cc:654] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.201962 11693 tablet_bootstrap.cc:654] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.201965 11691 tablet_bootstrap.cc:654] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.203085 11693 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: No bootstrap required, opened a new log
I20260811 04:37:02.203085 11691 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: No bootstrap required, opened a new log
I20260811 04:37:02.203164 11693 ts_tablet_manager.cc:1397] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:02.203164 11691 ts_tablet_manager.cc:1397] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:02.203193 11692 tablet_bootstrap.cc:492] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: No bootstrap required, opened a new log
I20260811 04:37:02.203277 11692 ts_tablet_manager.cc:1397] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:02.203581 11693 raft_consensus.cc:348] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.203581 11691 raft_consensus.cc:348] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.203701 11691 raft_consensus.cc:374] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.203701 11693 raft_consensus.cc:374] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.203744 11691 raft_consensus.cc:729] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.203748 11693 raft_consensus.cc:729] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.203752 11692 raft_consensus.cc:348] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.203882 11692 raft_consensus.cc:374] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.203878 11691 consensus_queue.cc:260] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.203871 11693 consensus_queue.cc:260] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.203931 11692 raft_consensus.cc:729] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.204066 11692 consensus_queue.cc:260] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.204135 11691 ts_tablet_manager.cc:1428] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:02.204135 11693 ts_tablet_manager.cc:1428] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 04:37:02.204319 11692 ts_tablet_manager.cc:1428] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:37:02.204380 11698 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.204530 11698 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.204147 11697 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.204569 11699 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.204677 11697 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.204681 11699 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.455595 11699 raft_consensus.cc:482] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:02.455750 11699 raft_consensus.cc:504] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.457031 11699 leader_election.cc:288] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099), 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069)
I20260811 04:37:02.461248 11737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.457326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:38099 (local address 127.2.238.129:56783)
0811 04:37:02.457644 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.457661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.457768 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.458222 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.458224 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.458245 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:02.458349 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.458354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.459249 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.459255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.460099 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.460109 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.460773 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.460889 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.460919 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.461008 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.461082 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":128,"threads_started":1}
I20260811 04:37:02.461248 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.457399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56783 (local address 127.2.238.130:38099)
0811 04:37:02.457714 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.457741 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.457766 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:37:02.457937 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.457940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.457997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.458093 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:02.458471 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.459134 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460225 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.460639 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460773 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.460835 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.461028 (+ 193us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.461101 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":89,"threads_started":1}
I20260811 04:37:02.461427 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.457943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:51307 (local address 127.2.238.131:40069)
0811 04:37:02.458093 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.458097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.458194 (+ 97us) server_negotiation.cc:407] Connection header received
0811 04:37:02.458341 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.458343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.458381 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.458463 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:02.458733 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.459348 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460445 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.460852 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460983 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.461038 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.461218 (+ 180us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.461292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 04:37:02.461542 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.457826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40069 (local address 127.2.238.129:51307)
0811 04:37:02.458124 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.458141 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.458234 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.458475 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.458477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.458493 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:02.458633 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.458640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.459470 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.459473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.460316 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.460334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.460975 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.460979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.461087 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.461113 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.461200 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.461269 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":61,"threads_started":1}
I20260811 04:37:02.461850 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911d1bf033f4e65a959dd11be28396e" candidate_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73" is_pre_election: true
I20260811 04:37:02.462018 11465 raft_consensus.cc:2454] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bbad433f3cf44adad9a5fa0e46398cd in term 0.
I20260811 04:37:02.462023 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911d1bf033f4e65a959dd11be28396e" candidate_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359ce486ef6a41ba9eb23da67c839dc2" is_pre_election: true
I20260811 04:37:02.462152 11614 raft_consensus.cc:2454] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bbad433f3cf44adad9a5fa0e46398cd in term 0.
I20260811 04:37:02.462424 11244 leader_election.cc:302] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fe18ced6234446a0dcce4b05ff4a73, 8bbad433f3cf44adad9a5fa0e46398cd; no voters:
I20260811 04:37:02.462742 11699 raft_consensus.cc:2781] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:02.462821 11699 raft_consensus.cc:482] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:02.462877 11699 raft_consensus.cc:3037] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.463929 11699 raft_consensus.cc:504] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.464277 11699 leader_election.cc:288] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [CANDIDATE]: Term 1 election: Requested vote from peers 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099), 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069)
I20260811 04:37:02.464680 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911d1bf033f4e65a959dd11be28396e" candidate_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73"
I20260811 04:37:02.464715 11614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f911d1bf033f4e65a959dd11be28396e" candidate_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "359ce486ef6a41ba9eb23da67c839dc2"
I20260811 04:37:02.464797 11465 raft_consensus.cc:3037] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.464846 11614 raft_consensus.cc:3037] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.465901 11465 raft_consensus.cc:2454] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bbad433f3cf44adad9a5fa0e46398cd in term 1.
I20260811 04:37:02.465916 11614 raft_consensus.cc:2454] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bbad433f3cf44adad9a5fa0e46398cd in term 1.
I20260811 04:37:02.466192 11244 leader_election.cc:302] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fe18ced6234446a0dcce4b05ff4a73, 8bbad433f3cf44adad9a5fa0e46398cd; no voters:
I20260811 04:37:02.466404 11699 raft_consensus.cc:2781] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:02.466609 11699 raft_consensus.cc:686] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 LEADER]: Becoming Leader. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Running, Role: LEADER
I20260811 04:37:02.466895 11699 consensus_queue.cc:237] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.468170 11740 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: TxnStatusTablet state changed. Reason: New leader 8bbad433f3cf44adad9a5fa0e46398cd. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.468318 11740 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:37:02.468648 11742 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260811 04:37:02.468811 11742 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260811 04:37:02.469555 11150 catalog_manager.cc:5495] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129). New cstate: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } health_report { overall_health: UNKNOWN } } }
I20260811 04:37:02.498881 11731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.494617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.0.0.1:55370)
0811 04:37:02.494821 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.494835 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.494925 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.495294 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.495296 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.495303 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:37:02.495396 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.495400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.496404 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.496406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.496994 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.497002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.497574 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.497577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.497697 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.497803 (+ 106us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.498573 (+ 770us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.498587 (+ 14us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.498645 (+ 58us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.498718 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 04:37:02.498859 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.494739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.2.238.129:45493)
0811 04:37:02.494880 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.494885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.494903 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:37:02.495082 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.495086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.495167 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.495273 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.495504 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.496290 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.497114 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.497467 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.497571 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.497956 (+ 385us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.498484 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.498594 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.498658 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.498714 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:02.502056 11465 raft_consensus.cc:1261] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Refusing update from remote peer 8bbad433f3cf44adad9a5fa0e46398cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.502249 11614 raft_consensus.cc:1261] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Refusing update from remote peer 8bbad433f3cf44adad9a5fa0e46398cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.502579 11740 consensus_queue.cc:1035] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.502782 11699 consensus_queue.cc:1035] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.504427 11698 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: TxnStatusTablet state changed. Reason: New leader 8bbad433f3cf44adad9a5fa0e46398cd. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.504561 11698 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.504575 11697 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: TxnStatusTablet state changed. Reason: New leader 8bbad433f3cf44adad9a5fa0e46398cd. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.504680 11697 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.507702 11740 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.507836 11740 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:37:02.507985 11740 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.508068 11740 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 8bbad433f3cf44adad9a5fa0e46398cd: This TxnStatusTablet replica's current role is: LEADER
I20260811 04:37:02.508050 11697 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.508162 11697 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 359ce486ef6a41ba9eb23da67c839dc2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.508252 11698 tablet_replica.cc:439] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } } }
I20260811 04:37:02.508355 11698 tablet_replica.cc:442] T f911d1bf033f4e65a959dd11be28396e P 52fe18ced6234446a0dcce4b05ff4a73: This TxnStatusTablet replica's current role is: FOLLOWER
I20260811 04:37:02.508738 11745 mvcc.cc:205] Tried to move back new op lower bound from 7317188700162928640 to 7317188700027158528. Current Snapshot: MvccSnapshot[applied={T|T < 7317188700162928640}]
I20260811 04:37:02.510883 11746 mvcc.cc:205] Tried to move back new op lower bound from 7317188700162928640 to 7317188700027158528. Current Snapshot: MvccSnapshot[applied={T|T < 7317188700162928640 or (T in {7317188700162928640})}]
I20260811 04:37:02.519347 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.515309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.2.238.130:38099)
0811 04:37:02.515426 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.515429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.515628 (+ 199us) server_negotiation.cc:407] Connection header received
0811 04:37:02.515848 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.515851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.515919 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.516037 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.516297 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.516912 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.517852 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.518070 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.518185 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.518482 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.518934 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.519068 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.519145 (+ 77us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.519203 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260811 04:37:02.531160 11759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.526454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.2.238.130:35965)
0811 04:37:02.526766 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.526789 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.526876 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.527148 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.527150 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.527156 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:37:02.527256 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.527260 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.528302 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.528305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.529762 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.529773 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.530406 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.530408 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.530494 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.530541 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.530868 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.530874 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.530928 (+ 54us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.530996 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":126,"threads_started":1}
I20260811 04:37:02.531149 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.526562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:35965 (local address 127.2.238.129:45493)
0811 04:37:02.526701 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.526706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.526860 (+ 154us) server_negotiation.cc:407] Connection header received
0811 04:37:02.526967 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.526970 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.527024 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.527106 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.527366 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.528142 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.529920 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.530282 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.530388 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.530692 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.530776 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.530847 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.530957 (+ 110us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.531009 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 04:37:02.538451 11614 raft_consensus.cc:1261] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Refusing update from remote peer 52fe18ced6234446a0dcce4b05ff4a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.538556 11315 raft_consensus.cc:1261] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Refusing update from remote peer 52fe18ced6234446a0dcce4b05ff4a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.538988 11698 consensus_queue.cc:1035] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.539175 11724 consensus_queue.cc:1035] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.563931 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.558985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:34513 (local address 127.2.238.130:38099)
0811 04:37:02.559093 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.559097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.559111 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:02.559192 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.559194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.559239 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.559299 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.559616 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.560377 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.562635 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.562952 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.563020 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.563458 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.563538 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.563580 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.563736 (+ 156us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.563789 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260811 04:37:02.564368 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.558581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:38099 (local address 127.2.238.129:34513)
0811 04:37:02.558957 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.558972 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.559106 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.559333 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.559336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.559344 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:37:02.559481 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.559487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.560507 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.560511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.562488 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.562501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.563157 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.563160 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.563253 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.563302 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.563643 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.563648 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.564145 (+ 497us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.564229 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":145,"threads_started":1}
I20260811 04:37:02.591717 11151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:02.591908 11151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46718:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:02.597813 11151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Sending DeleteTablet for 3 replicas of tablet c7ab70242a81412cb1d5709881a792df
I20260811 04:37:02.598492 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ab70242a81412cb1d5709881a792df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49392
I20260811 04:37:02.598619 11444 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ab70242a81412cb1d5709881a792df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:44452
I20260811 04:37:02.599290 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ab70242a81412cb1d5709881a792df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55364
I20260811 04:37:02.605636 11773 tablet_replica.cc:330] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: stopping tablet replica
I20260811 04:37:02.605868 11773 raft_consensus.cc:2229] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:02.606077 11773 raft_consensus.cc:2258] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.606963 11773 ts_tablet_manager.cc:1905] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.609968 11771 tablet_replica.cc:330] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: stopping tablet replica
I20260811 04:37:02.610101 11771 raft_consensus.cc:2229] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:02.610219 11771 raft_consensus.cc:2258] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.610663 11772 tablet_replica.cc:330] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: stopping tablet replica
I20260811 04:37:02.610795 11772 raft_consensus.cc:2229] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:37:02.610795 11773 ts_tablet_manager.cc:1918] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:37:02.610877 11771 ts_tablet_manager.cc:1905] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.610996 11773 log.cc:1196] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/c7ab70242a81412cb1d5709881a792df
I20260811 04:37:02.611125 11772 raft_consensus.cc:2258] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.611449 11773 ts_tablet_manager.cc:1939] T c7ab70242a81412cb1d5709881a792df P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting consensus metadata
I20260811 04:37:02.611758 11772 ts_tablet_manager.cc:1905] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.612375 11134 catalog_manager.cc:4841] TS 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493): tablet c7ab70242a81412cb1d5709881a792df (table test-table [id=fd5061b799ae431b9aa40e63497e37cf]) successfully deleted
I20260811 04:37:02.613725 11771 ts_tablet_manager.cc:1918] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:37:02.613830 11771 log.cc:1196] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/c7ab70242a81412cb1d5709881a792df
I20260811 04:37:02.614094 11771 ts_tablet_manager.cc:1939] T c7ab70242a81412cb1d5709881a792df P 359ce486ef6a41ba9eb23da67c839dc2: Deleting consensus metadata
I20260811 04:37:02.614750 11134 catalog_manager.cc:4841] TS 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069): tablet c7ab70242a81412cb1d5709881a792df (table test-table [id=fd5061b799ae431b9aa40e63497e37cf]) successfully deleted
I20260811 04:37:02.614868 11772 ts_tablet_manager.cc:1918] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:37:02.614928 11772 log.cc:1196] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/c7ab70242a81412cb1d5709881a792df
I20260811 04:37:02.615192 11772 ts_tablet_manager.cc:1939] T c7ab70242a81412cb1d5709881a792df P 52fe18ced6234446a0dcce4b05ff4a73: Deleting consensus metadata
I20260811 04:37:02.617967 11134 catalog_manager.cc:4841] TS 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099): tablet c7ab70242a81412cb1d5709881a792df (table test-table [id=fd5061b799ae431b9aa40e63497e37cf]) successfully deleted
Aug 11 04:37:02 dist-test-slave-0kls krb5kdc[11103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 11 04:37:02 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:02 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260811 04:37:02.645678 11732 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:02.637667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46730 (local address 127.2.238.190:38285)
0811 04:37:02.637865 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.637870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.638067 (+ 197us) server_negotiation.cc:407] Connection header received
0811 04:37:02.638206 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.638209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.638266 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.638354 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:02.639280 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.640127 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.641370 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.641747 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.641879 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.645540 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46730: BlockingRecv error: recv got EOF from 127.0.0.1:46730 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 04:37:02 dist-test-slave-0kls krb5kdc[11103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM
I20260811 04:37:02.656878 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.647100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.2.238.190:38285)
0811 04:37:02.647223 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.647227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.647620 (+ 393us) server_negotiation.cc:407] Connection header received
0811 04:37:02.647702 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.647707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.647768 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.647854 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:02.648505 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.649287 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.650420 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.650766 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.650869 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.653458 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:02.653461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:02.653464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:02.653488 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:02.655056 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:02.655521 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:02.655524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:02.655526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:02.655609 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:02.655877 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:02.655880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:02.655881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:02.656112 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:02.656240 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.656542 (+ 302us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.656662 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:37:02.659646 11151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:37:02.660132 11151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:02.664951 11444 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b (DEFAULT_TABLE table=test-table [id=afd2804a105a45829ef15bb81be4632d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.665169 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b (DEFAULT_TABLE table=test-table [id=afd2804a105a45829ef15bb81be4632d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.665196 11294 tablet_service.cc:1467] Processing CreateTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b (DEFAULT_TABLE table=test-table [id=afd2804a105a45829ef15bb81be4632d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.665306 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab115053b7b4fa3baf3bde92036ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.665423 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab115053b7b4fa3baf3bde92036ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.665473 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ab115053b7b4fa3baf3bde92036ed4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.667853 11693 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Bootstrap starting.
I20260811 04:37:02.667929 11692 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Bootstrap starting.
I20260811 04:37:02.668705 11693 tablet_bootstrap.cc:654] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.668741 11692 tablet_bootstrap.cc:654] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.669238 11691 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Bootstrap starting.
I20260811 04:37:02.669894 11693 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: No bootstrap required, opened a new log
I20260811 04:37:02.669984 11693 ts_tablet_manager.cc:1397] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:02.670531 11691 tablet_bootstrap.cc:654] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.670444 11693 raft_consensus.cc:348] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.670589 11693 raft_consensus.cc:374] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.670634 11693 raft_consensus.cc:729] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.670780 11693 consensus_queue.cc:260] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.671073 11693 ts_tablet_manager.cc:1428] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:02.671905 11691 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: No bootstrap required, opened a new log
I20260811 04:37:02.671984 11691 ts_tablet_manager.cc:1397] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260811 04:37:02.672447 11691 raft_consensus.cc:348] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.672556 11691 raft_consensus.cc:374] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.672597 11691 raft_consensus.cc:729] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.672729 11691 consensus_queue.cc:260] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.672998 11691 ts_tablet_manager.cc:1428] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:02.673979 11692 tablet_bootstrap.cc:492] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: No bootstrap required, opened a new log
I20260811 04:37:02.674086 11692 ts_tablet_manager.cc:1397] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20260811 04:37:02.674508 11692 raft_consensus.cc:348] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.674623 11692 raft_consensus.cc:374] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.674661 11692 raft_consensus.cc:729] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.674821 11692 consensus_queue.cc:260] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.675107 11692 ts_tablet_manager.cc:1428] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:02.878952 11697 raft_consensus.cc:482] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:02.879086 11697 raft_consensus.cc:504] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.879477 11697 leader_election.cc:288] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099)
I20260811 04:37:02.879978 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab115053b7b4fa3baf3bde92036ed4b" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" is_pre_election: true
I20260811 04:37:02.880039 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab115053b7b4fa3baf3bde92036ed4b" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73" is_pre_election: true
I20260811 04:37:02.880158 11315 raft_consensus.cc:2454] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:02.880221 11465 raft_consensus.cc:2454] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:02.880571 11544 leader_election.cc:302] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 8bbad433f3cf44adad9a5fa0e46398cd; no voters:
I20260811 04:37:02.880765 11697 raft_consensus.cc:2781] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:02.880833 11697 raft_consensus.cc:482] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:02.880868 11697 raft_consensus.cc:3037] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.882166 11697 raft_consensus.cc:504] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.882519 11697 leader_election.cc:288] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099)
I20260811 04:37:02.882840 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab115053b7b4fa3baf3bde92036ed4b" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
I20260811 04:37:02.882854 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ab115053b7b4fa3baf3bde92036ed4b" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73"
I20260811 04:37:02.882937 11315 raft_consensus.cc:3037] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.882983 11465 raft_consensus.cc:3037] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.883937 11465 raft_consensus.cc:2454] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:02.883945 11315 raft_consensus.cc:2454] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:02.884207 11544 leader_election.cc:302] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 52fe18ced6234446a0dcce4b05ff4a73; no voters:
I20260811 04:37:02.884371 11697 raft_consensus.cc:2781] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:02.884560 11697 raft_consensus.cc:686] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Running, Role: LEADER
I20260811 04:37:02.884718 11697 consensus_queue.cc:237] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.886399 11150 catalog_manager.cc:5495] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "359ce486ef6a41ba9eb23da67c839dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } health_report { overall_health: HEALTHY } } }
I20260811 04:37:02.909175 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.906396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49410 (local address 127.2.238.131:40069)
0811 04:37:02.906528 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.906531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.906544 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:02.906625 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.906628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.906667 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.906729 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.906965 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.907431 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.908183 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.908385 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.908463 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.908798 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.908861 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.908913 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.909032 (+ 119us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.909076 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:02.911248 11465 raft_consensus.cc:1261] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.911242 11315 raft_consensus.cc:1261] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.911688 11792 consensus_queue.cc:1035] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.911835 11697 consensus_queue.cc:1035] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.920013 11150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:02.920176 11150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:02.922061 11150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Sending DeleteTablet for 3 replicas of tablet 5ab115053b7b4fa3baf3bde92036ed4b
I20260811 04:37:02.922570 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55364
I20260811 04:37:02.922640 11444 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:44452
I20260811 04:37:02.922639 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ab115053b7b4fa3baf3bde92036ed4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:02 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49392
I20260811 04:37:02.922765 11773 tablet_replica.cc:330] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: stopping tablet replica
I20260811 04:37:02.922822 11771 tablet_replica.cc:330] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: stopping tablet replica
I20260811 04:37:02.922930 11773 raft_consensus.cc:2229] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:02.922945 11771 raft_consensus.cc:2229] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:37:02.923096 11773 raft_consensus.cc:2258] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.923108 11771 raft_consensus.cc:2258] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.923321 11772 tablet_replica.cc:330] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: stopping tablet replica
I20260811 04:37:02.923471 11772 raft_consensus.cc:2229] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:02.923597 11772 raft_consensus.cc:2258] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:02.923804 11773 ts_tablet_manager.cc:1905] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.923923 11771 ts_tablet_manager.cc:1905] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.924273 11772 ts_tablet_manager.cc:1905] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:02.926378 11771 ts_tablet_manager.cc:1918] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:02.926378 11773 ts_tablet_manager.cc:1918] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:02.926384 11772 ts_tablet_manager.cc:1918] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:02.926460 11771 log.cc:1196] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/5ab115053b7b4fa3baf3bde92036ed4b
I20260811 04:37:02.926460 11773 log.cc:1196] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/5ab115053b7b4fa3baf3bde92036ed4b
I20260811 04:37:02.926486 11772 log.cc:1196] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/5ab115053b7b4fa3baf3bde92036ed4b
I20260811 04:37:02.926728 11771 ts_tablet_manager.cc:1939] T 5ab115053b7b4fa3baf3bde92036ed4b P 359ce486ef6a41ba9eb23da67c839dc2: Deleting consensus metadata
I20260811 04:37:02.926748 11773 ts_tablet_manager.cc:1939] T 5ab115053b7b4fa3baf3bde92036ed4b P 8bbad433f3cf44adad9a5fa0e46398cd: Deleting consensus metadata
I20260811 04:37:02.926764 11772 ts_tablet_manager.cc:1939] T 5ab115053b7b4fa3baf3bde92036ed4b P 52fe18ced6234446a0dcce4b05ff4a73: Deleting consensus metadata
I20260811 04:37:02.927361 11134 catalog_manager.cc:4841] TS 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493): tablet 5ab115053b7b4fa3baf3bde92036ed4b (table test-table [id=afd2804a105a45829ef15bb81be4632d]) successfully deleted
I20260811 04:37:02.927516 11134 catalog_manager.cc:4841] TS 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069): tablet 5ab115053b7b4fa3baf3bde92036ed4b (table test-table [id=afd2804a105a45829ef15bb81be4632d]) successfully deleted
I20260811 04:37:02.927659 11134 catalog_manager.cc:4841] TS 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099): tablet 5ab115053b7b4fa3baf3bde92036ed4b (table test-table [id=afd2804a105a45829ef15bb81be4632d]) successfully deleted
I20260811 04:37:02.928028 11150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:37:02.928462 11150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:02.932766 11294 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 (DEFAULT_TABLE table=test-table [id=8538365ff0c04f6f8ddbee036a8fb946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.932847 11444 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 (DEFAULT_TABLE table=test-table [id=8538365ff0c04f6f8ddbee036a8fb946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.933090 11294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ee2699a104e35b34e9b92c926e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.933109 11444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ee2699a104e35b34e9b92c926e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.934842 11691 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Bootstrap starting.
I20260811 04:37:02.934978 11692 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: Bootstrap starting.
I20260811 04:37:02.935055 11593 tablet_service.cc:1467] Processing CreateTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 (DEFAULT_TABLE table=test-table [id=8538365ff0c04f6f8ddbee036a8fb946]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:02.935276 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c8ee2699a104e35b34e9b92c926e442. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:02.935698 11691 tablet_bootstrap.cc:654] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.935717 11692 tablet_bootstrap.cc:654] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.936796 11692 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: No bootstrap required, opened a new log
I20260811 04:37:02.936874 11692 ts_tablet_manager.cc:1397] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:02.937184 11691 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: No bootstrap required, opened a new log
I20260811 04:37:02.937247 11691 ts_tablet_manager.cc:1397] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:02.937227 11692 raft_consensus.cc:348] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.937335 11692 raft_consensus.cc:374] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.937369 11692 raft_consensus.cc:729] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bbad433f3cf44adad9a5fa0e46398cd, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.937489 11693 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Bootstrap starting.
I20260811 04:37:02.937480 11692 consensus_queue.cc:260] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.937644 11691 raft_consensus.cc:348] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.937745 11691 raft_consensus.cc:374] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.937754 11692 ts_tablet_manager.cc:1428] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260811 04:37:02.937779 11691 raft_consensus.cc:729] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fe18ced6234446a0dcce4b05ff4a73, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.937935 11691 consensus_queue.cc:260] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.938251 11691 ts_tablet_manager.cc:1428] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:37:02.938441 11693 tablet_bootstrap.cc:654] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:02.939558 11693 tablet_bootstrap.cc:492] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: No bootstrap required, opened a new log
I20260811 04:37:02.939649 11693 ts_tablet_manager.cc:1397] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:02.940012 11693 raft_consensus.cc:348] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.940150 11693 raft_consensus.cc:374] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:02.940521 11693 raft_consensus.cc:729] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Initialized, Role: FOLLOWER
I20260811 04:37:02.940912 11693 consensus_queue.cc:260] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.941211 11693 ts_tablet_manager.cc:1428] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:02.959934 11792 raft_consensus.cc:482] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:02.960029 11792 raft_consensus.cc:504] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.960417 11792 leader_election.cc:288] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099)
I20260811 04:37:02.960798 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ee2699a104e35b34e9b92c926e442" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73" is_pre_election: true
I20260811 04:37:02.960798 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ee2699a104e35b34e9b92c926e442" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" is_pre_election: true
I20260811 04:37:02.960947 11465 raft_consensus.cc:2454] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:02.960969 11315 raft_consensus.cc:2454] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 0.
I20260811 04:37:02.961211 11544 leader_election.cc:302] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 52fe18ced6234446a0dcce4b05ff4a73; no voters:
I20260811 04:37:02.961374 11792 raft_consensus.cc:2781] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:02.961463 11792 raft_consensus.cc:482] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:02.961526 11792 raft_consensus.cc:3037] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.962411 11792 raft_consensus.cc:504] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.962723 11792 leader_election.cc:288] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493), 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099)
I20260811 04:37:02.963003 11315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ee2699a104e35b34e9b92c926e442" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bbad433f3cf44adad9a5fa0e46398cd"
I20260811 04:37:02.963023 11465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c8ee2699a104e35b34e9b92c926e442" candidate_uuid: "359ce486ef6a41ba9eb23da67c839dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fe18ced6234446a0dcce4b05ff4a73"
I20260811 04:37:02.963116 11315 raft_consensus.cc:3037] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.963115 11465 raft_consensus.cc:3037] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:02.964069 11315 raft_consensus.cc:2454] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:02.964095 11465 raft_consensus.cc:2454] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359ce486ef6a41ba9eb23da67c839dc2 in term 1.
I20260811 04:37:02.964323 11544 leader_election.cc:302] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359ce486ef6a41ba9eb23da67c839dc2, 8bbad433f3cf44adad9a5fa0e46398cd; no voters:
I20260811 04:37:02.964473 11792 raft_consensus.cc:2781] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:02.964550 11792 raft_consensus.cc:686] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Becoming Leader. State: Replica: 359ce486ef6a41ba9eb23da67c839dc2, State: Running, Role: LEADER
I20260811 04:37:02.964699 11792 consensus_queue.cc:237] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } }
I20260811 04:37:02.966115 11150 catalog_manager.cc:5495] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "359ce486ef6a41ba9eb23da67c839dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359ce486ef6a41ba9eb23da67c839dc2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 40069 } health_report { overall_health: HEALTHY } } }
I20260811 04:37:02.993345 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.990476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:39147 (local address 127.2.238.129:45493)
0811 04:37:02.990645 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:02.990649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:02.990737 (+ 88us) server_negotiation.cc:407] Connection header received
0811 04:37:02.990842 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:02.990844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:02.990885 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:02.990948 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:02.991149 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.991722 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.992495 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.992714 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.992787 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.993022 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.993081 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.993135 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:02.993203 (+ 68us) server_negotiation.cc:299] Negotiation successful
0811 04:37:02.993239 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 04:37:02.993371 11805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:02.990374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.2.238.131:39147)
0811 04:37:02.990660 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:02.990673 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:02.990762 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:02.990968 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:02.990970 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:02.990976 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:37:02.991061 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.991066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.991808 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.991810 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.992404 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:02.992412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:02.992804 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:02.992806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:02.992864 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:02.992907 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:37:02.993149 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:37:02.993153 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:37:02.993196 (+ 43us) client_negotiation.cc:240] Negotiation successful
0811 04:37:02.993261 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":88,"threads_started":1}
I20260811 04:37:02.998454 11465 raft_consensus.cc:1261] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.998454 11315 raft_consensus.cc:1261] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Refusing update from remote peer 359ce486ef6a41ba9eb23da67c839dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:02.998812 11697 consensus_queue.cc:1035] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bbad433f3cf44adad9a5fa0e46398cd" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 45493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:02.998974 11792 consensus_queue.cc:1035] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fe18ced6234446a0dcce4b05ff4a73" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 38099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:03.015913 11690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:03.013091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:43875 (local address 127.2.238.131:40069)
0811 04:37:03.013195 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:03.013199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:03.013212 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:03.013258 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:03.013260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:03.013305 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:03.013371 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:03.013544 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:03.013973 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:03.014694 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:03.014980 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:03.015077 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:03.015438 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:03.015518 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:03.015591 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:03.015713 (+ 122us) server_negotiation.cc:299] Negotiation successful
0811 04:37:03.015759 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260811 04:37:03.015928 11766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:03.012931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:40069 (local address 127.2.238.129:43875)
0811 04:37:03.013049 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:03.013062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:03.013162 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:03.013392 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:03.013394 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:03.013400 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0811 04:37:03.013476 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:03.013480 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:03.014047 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:03.014049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:03.014581 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:03.014590 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:03.015147 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:03.015151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:03.015260 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:03.015295 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0811 04:37:03.015618 (+ 323us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0811 04:37:03.015626 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:37:03.015709 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:37:03.015807 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260811 04:37:03.030949 11150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:03.031082 11150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:03.032840 11150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1030e848120e43b9b4ae0f3a2c3ab996: Sending DeleteTablet for 3 replicas of tablet 1c8ee2699a104e35b34e9b92c926e442
I20260811 04:37:03.033324 11294 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:03 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:55364
I20260811 04:37:03.033463 11444 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:03 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:44452
I20260811 04:37:03.033574 11773 tablet_replica.cc:330] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd: stopping tablet replica
I20260811 04:37:03.033672 11772 tablet_replica.cc:330] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: stopping tablet replica
I20260811 04:37:03.033663 11593 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c8ee2699a104e35b34e9b92c926e442 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:03 UTC) from {username='oryx', principal='oryx/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49392
I20260811 04:37:03.033730 11773 raft_consensus.cc:2229] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:03.033831 11772 raft_consensus.cc:2229] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:03.033862 11771 tablet_replica.cc:330] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: stopping tablet replica
I20260811 04:37:03.033865 11773 raft_consensus.cc:2258] T 1c8ee2699a104e35b34e9b92c926e442 P 8bbad433f3cf44adad9a5fa0e46398cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:03.033932 11772 raft_consensus.cc:2258] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:03.033977 11771 raft_consensus.cc:2229] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:37:03.034160 11771 raft_consensus.cc:2258] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:03.034438 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11226
I20260811 04:37:03.034720 11772 ts_tablet_manager.cc:1905] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:03.034775 11771 ts_tablet_manager.cc:1905] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:03.037196 11772 ts_tablet_manager.cc:1918] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:37:03.037196 11771 ts_tablet_manager.cc:1918] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260811 04:37:03.037263 11772 log.cc:1196] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/1c8ee2699a104e35b34e9b92c926e442
I20260811 04:37:03.037271 11771 log.cc:1196] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/1c8ee2699a104e35b34e9b92c926e442
I20260811 04:37:03.037535 11771 ts_tablet_manager.cc:1939] T 1c8ee2699a104e35b34e9b92c926e442 P 359ce486ef6a41ba9eb23da67c839dc2: Deleting consensus metadata
I20260811 04:37:03.037535 11772 ts_tablet_manager.cc:1939] T 1c8ee2699a104e35b34e9b92c926e442 P 52fe18ced6234446a0dcce4b05ff4a73: Deleting consensus metadata
I20260811 04:37:03.038568 11134 catalog_manager.cc:4841] TS 359ce486ef6a41ba9eb23da67c839dc2 (127.2.238.131:40069): tablet 1c8ee2699a104e35b34e9b92c926e442 (table test-table [id=8538365ff0c04f6f8ddbee036a8fb946]) successfully deleted
I20260811 04:37:03.042795 11134 catalog_manager.cc:4841] TS 52fe18ced6234446a0dcce4b05ff4a73 (127.2.238.130:38099): tablet 1c8ee2699a104e35b34e9b92c926e442 (table test-table [id=8538365ff0c04f6f8ddbee036a8fb946]) successfully deleted
W20260811 04:37:03.050151 11134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:45493 (error 108)
I20260811 04:37:03.050447 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11374
I20260811 04:37:03.052860 11811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:03.052283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:45493 (local address 127.0.0.1:55380)
0811 04:37:03.052658 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:03.052747 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:45493: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":48,"threads_started":1}
W20260811 04:37:03.052955 11134 catalog_manager.cc:4584] TS 8bbad433f3cf44adad9a5fa0e46398cd (127.2.238.129:45493): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c8ee2699a104e35b34e9b92c926e442: Network error: Client connection negotiation failed: client connection to 127.2.238.129:45493: connect: Connection refused (error 111)
I20260811 04:37:03.060423 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11526
I20260811 04:37:03.068389 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11117
2026-08-11T04:37:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3636 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:37:03 dist-test-slave-0kls krb5kdc[11814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:37:03 dist-test-slave-0kls krb5kdc[11814](info): set up 2 sockets
Aug 11 04:37:03 dist-test-slave-0kls krb5kdc[11814](info): commencing operation
krb5kdc: starting...
W20260811 04:37:04.118047 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:37:04 dist-test-slave-0kls krb5kdc[11814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:37:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:37:04.244661 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40665
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:40665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:04.355142 11829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:04.355386 11829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:04.355420 11829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:04.359212 11829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:04.359261 11829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:04.359284 11829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:04.359301 11829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:04.359315 11829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:04.364286 11829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:40665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:04.365221 11829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:04.366290 11829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:04.374697 11835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:04.374703 11837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:04.374742 11834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:04.374909 11829 server_base.cc:1034] running on GCE node
I20260811 04:37:04.375581 11829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:04.376652 11829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:04.377851 11829 hybrid_clock.cc:648] HybridClock initialized: now 1786423024377837 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-11T04:37:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1309 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:37:04 dist-test-slave-0kls krb5kdc[11843](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:37:04 dist-test-slave-0kls krb5kdc[11843](info): set up 2 sockets
Aug 11 04:37:04 dist-test-slave-0kls krb5kdc[11843](info): commencing operation
krb5kdc: starting...
W20260811 04:37:05.405025 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:37:05 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:37:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:37:05.528378 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40775
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:05.638789 11857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:05.639053 11857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:05.639097 11857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:05.643014 11857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:05.643067 11857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:05.643093 11857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:05.643110 11857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:05.643178 11857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:05.648756 11857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:05.649739 11857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:05.650789 11857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:05.658535 11866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:05.658572 11864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:05.658576 11863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:05.658677 11857 server_base.cc:1034] running on GCE node
I20260811 04:37:05.659232 11857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:05.660262 11857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:05.661427 11857 hybrid_clock.cc:648] HybridClock initialized: now 1786423025661398 us; error 41 us; skew 500 ppm
Aug 11 04:37:05 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:05.664095 11857 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:37:05.665009 11857 webserver.cc:460] Webserver started at http://127.2.238.190:41285/ using document root <none> and password file <none>
I20260811 04:37:05.665582 11857 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:05.665673 11857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:05.665952 11857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:05.667865 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "0756dd4d1be44d14bb9c997c52e87f41"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "f16095c26f43c67716b9b183214cdefb"
server_key_iv: "a8001a2c8ecb1d02f06334630a28b7d7"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.668401 11857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "0756dd4d1be44d14bb9c997c52e87f41"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "f16095c26f43c67716b9b183214cdefb"
server_key_iv: "a8001a2c8ecb1d02f06334630a28b7d7"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.672020 11857 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:37:05.674471 11873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:05.675645 11857 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:05.675774 11857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "0756dd4d1be44d14bb9c997c52e87f41"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "f16095c26f43c67716b9b183214cdefb"
server_key_iv: "a8001a2c8ecb1d02f06334630a28b7d7"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.675876 11857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:05.702484 11857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:05.705327 11857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:05.705521 11857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:05.712577 11857 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:40775
I20260811 04:37:05.712594 11929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:40775 every 8 connection(s)
I20260811 04:37:05.713450 11857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:05.714506 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11857
I20260811 04:37:05.714649 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:37:05.714954 3002 external_mini_cluster.cc:1428] Setting key db4abfe84569ec5d3c939ba90b66f4d1
I20260811 04:37:05.717180 11930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:05.722944 11930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Bootstrap starting.
Aug 11 04:37:05 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:05.725565 11930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:05.726341 11930 log.cc:824] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:05.728439 11930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: No bootstrap required, opened a new log
I20260811 04:37:05.729188 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:05.716190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.2.238.190:40775)
0811 04:37:05.716636 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:05.716652 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:05.716690 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:37:05.717384 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:05.717402 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:05.717737 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:05.718134 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:05.718968 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:05.719956 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:05.720877 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:05.721411 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:05.721620 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:05.724491 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:05.724496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:05.724508 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:05.724557 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:05.726272 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:05.726755 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:05.726760 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:05.726764 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:05.726878 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:05.727203 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:05.727206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:05.727207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:05.727637 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:05.727815 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:05.728074 (+ 259us) server_negotiation.cc:299] Negotiation successful
0811 04:37:05.728346 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260811 04:37:05.732983 11930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:05.733268 11930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:05.733398 11930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0756dd4d1be44d14bb9c997c52e87f41, State: Initialized, Role: FOLLOWER
I20260811 04:37:05.733863 11930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:05.734045 11930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:05.734134 11930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:05.734234 11930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:05.735112 11930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:05.735455 11930 leader_election.cc:302] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0756dd4d1be44d14bb9c997c52e87f41; no voters:
I20260811 04:37:05.735769 11930 leader_election.cc:288] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:37:05.735920 11935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:05.736177 11935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 LEADER]: Becoming Leader. State: Replica: 0756dd4d1be44d14bb9c997c52e87f41, State: Running, Role: LEADER
I20260811 04:37:05.736563 11935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:05.736951 11930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:05.738322 11937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0756dd4d1be44d14bb9c997c52e87f41. Latest consensus state: current_term: 1 leader_uuid: "0756dd4d1be44d14bb9c997c52e87f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } } }
I20260811 04:37:05.738338 11936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0756dd4d1be44d14bb9c997c52e87f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } } }
I20260811 04:37:05.738511 11936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:05.738900 11945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:05.739341 11937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:05.741654 11945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:05.746634 11945 catalog_manager.cc:1347] Generated new cluster ID: b2a560ad148847ea946f2be216920da3
I20260811 04:37:05.746690 11945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:05.759717 11945 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:37:05.760689 11945 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:05.776714 11945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Generated new TSK 0
I20260811 04:37:05.777407 11945 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:37:05.816256 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:05.926190 11956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:05.926447 11956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:05.926514 11956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:05.930399 11956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:05.930451 11956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:05.930573 11956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:05.935887 11956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:05.936962 11956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:05.938190 11956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:37:05.948243 11956 server_base.cc:1034] running on GCE node
W20260811 04:37:05.948249 11962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:05.948223 11961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:05.948276 11964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:05.948880 11956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:05.949517 11956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:05.950681 11956 hybrid_clock.cc:648] HybridClock initialized: now 1786423025950655 us; error 37 us; skew 500 ppm
Aug 11 04:37:05 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:05.953428 11956 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:37:05.954355 11956 webserver.cc:460] Webserver started at http://127.2.238.129:41063/ using document root <none> and password file <none>
I20260811 04:37:05.954929 11956 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:05.955021 11956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:05.955257 11956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:05.957289 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "20d8711efe8746c8a3f0f7cfb008fd92"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "42d03b44d055c2720ef173ac388d75d0"
server_key_iv: "4b44bef940e4aec4c87312f21867b140"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.957921 11956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "20d8711efe8746c8a3f0f7cfb008fd92"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "42d03b44d055c2720ef173ac388d75d0"
server_key_iv: "4b44bef940e4aec4c87312f21867b140"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.961685 11956 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:37:05.964018 11971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:05.965178 11956 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:05.965322 11956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "20d8711efe8746c8a3f0f7cfb008fd92"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "42d03b44d055c2720ef173ac388d75d0"
server_key_iv: "4b44bef940e4aec4c87312f21867b140"
server_key_version: "encryptionkey@0"
I20260811 04:37:05.965435 11956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:05.990299 11956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:05.993387 11956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:05.993577 11956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:05.994257 11956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:05.995146 11956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:05.995224 11956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:05.995309 11956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:05.995362 11956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.004720 11956 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:39085
I20260811 04:37:06.004771 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:39085 every 8 connection(s)
I20260811 04:37:06.005846 11956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:06.012432 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 11956
I20260811 04:37:06.012559 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:37:06.012858 3002 external_mini_cluster.cc:1428] Setting key 68fa116efa7fe85824db598612a75ffa
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423025, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:06.019865 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.009335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:34339 (local address 127.2.238.190:40775)
0811 04:37:06.009481 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.009484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.009497 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:06.010334 (+ 837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.010336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.010379 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.010446 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.011550 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.012057 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.013232 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.013467 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.013574 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.016224 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:06.016227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:06.016230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:06.016263 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:06.017690 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.018247 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.018252 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.018255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.018339 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.018672 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.018675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.018676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.018859 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:06.018972 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:06.019609 (+ 637us) server_negotiation.cc:299] Negotiation successful
0811 04:37:06.019716 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 04:37:06.021142 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.008306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:34339)
0811 04:37:06.008854 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.008895 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.010139 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.010590 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.010596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.010957 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.011422 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.011435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.012174 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.012176 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.013118 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.013127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.013643 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.013646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.013780 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.014442 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.014461 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.016061 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:06.017867 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.017872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.017886 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.018127 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.018430 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.018433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.018435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.018577 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.019010 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:06.019019 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:06.019354 (+ 335us) client_negotiation.cc:769] Sending connection context
0811 04:37:06.019559 (+ 205us) client_negotiation.cc:240] Negotiation successful
0811 04:37:06.019947 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":145,"threads_started":1}
I20260811 04:37:06.022527 12087 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40775
I20260811 04:37:06.022770 12087 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:06.023255 12087 heartbeater.cc:511] Master 127.2.238.190:40775 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:37:06.024893 11891 ts_manager.cc:194] Registered new tserver with Master: 20d8711efe8746c8a3f0f7cfb008fd92 (127.2.238.129:39085)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:37:06.026274 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:34339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:37:06.042295 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:06.155179 12092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:06.155424 12092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:06.155503 12092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:06.159474 12092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:06.159538 12092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:06.159664 12092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:06.164813 12092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:06.165827 12092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:06.166875 12092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:06.179818 12100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.179863 12097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.179847 12098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:06.180140 12092 server_base.cc:1034] running on GCE node
I20260811 04:37:06.180621 12092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:06.181244 12092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:06.182438 12092 hybrid_clock.cc:648] HybridClock initialized: now 1786423026182419 us; error 31 us; skew 500 ppm
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:06.189286 12092 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:37:06.190243 12092 webserver.cc:460] Webserver started at http://127.2.238.130:40831/ using document root <none> and password file <none>
I20260811 04:37:06.190832 12092 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:06.190927 12092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:06.191167 12092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:06.193224 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "f260c4639aab441e87cc5bab4ae6702f"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "663158af3509712a4e20639da519c198"
server_key_iv: "2013e7227c1941915e37e411e3ccb905"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.193862 12092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "f260c4639aab441e87cc5bab4ae6702f"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "663158af3509712a4e20639da519c198"
server_key_iv: "2013e7227c1941915e37e411e3ccb905"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.197479 12092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:37:06.199983 12107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.201169 12092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:06.201324 12092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "f260c4639aab441e87cc5bab4ae6702f"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "663158af3509712a4e20639da519c198"
server_key_iv: "2013e7227c1941915e37e411e3ccb905"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.201438 12092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:06.219436 12092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:06.222113 12092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:06.222318 12092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:06.222841 12092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:06.223644 12092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:06.223714 12092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.223783 12092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:06.223827 12092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.232645 12092 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:46881
I20260811 04:37:06.232668 12230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:46881 every 8 connection(s)
I20260811 04:37:06.233412 12092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:06.238761 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12092
I20260811 04:37:06.238881 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:37:06.239146 3002 external_mini_cluster.cc:1428] Setting key 4c1b72851f235b00640a49b78f33ebb2
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:06.247059 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.235034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41545 (local address 127.2.238.190:40775)
0811 04:37:06.235183 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.235188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.235922 (+ 734us) server_negotiation.cc:407] Connection header received
0811 04:37:06.237127 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.237130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.237176 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.237244 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.238533 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.239316 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.240641 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.240961 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.241086 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.243574 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:06.243578 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:06.243581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:06.243606 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:06.245004 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.245563 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.245567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.245568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.245648 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.246018 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.246021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.246023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.246190 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:06.246320 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:06.246787 (+ 467us) server_negotiation.cc:299] Negotiation successful
0811 04:37:06.246904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:37:06.247797 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.235374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:41545)
0811 04:37:06.235785 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.235823 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.236905 (+ 1082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.237408 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.237414 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.237895 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.238394 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.238407 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.239442 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.239445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.240542 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.240552 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.241136 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.241138 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.241268 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.241873 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.241888 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.243420 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:06.245137 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.245143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.245156 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.245458 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.245737 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.245740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.245741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.245919 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.246320 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:06.246327 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:06.246550 (+ 223us) client_negotiation.cc:769] Sending connection context
0811 04:37:06.246721 (+ 171us) client_negotiation.cc:240] Negotiation successful
0811 04:37:06.247007 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":125,"threads_started":1}
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:37:06.248912 12231 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40775
I20260811 04:37:06.249130 12231 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:06.249575 12231 heartbeater.cc:511] Master 127.2.238.190:40775 requested a full tablet report, sending...
I20260811 04:37:06.250670 11891 ts_manager.cc:194] Registered new tserver with Master: f260c4639aab441e87cc5bab4ae6702f (127.2.238.130:46881)
I20260811 04:37:06.251307 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:41545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:37:06.267025 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:06.374897 12236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:06.375118 12236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:06.375154 12236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:06.378928 12236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:06.378973 12236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:06.379058 12236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:06.383908 12236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:06.384766 12236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:06.385732 12236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:06.395203 12241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.395196 12244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:06.395287 12236 server_base.cc:1034] running on GCE node
W20260811 04:37:06.395196 12242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:06.395800 12236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:06.396334 12236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:06.397508 12236 hybrid_clock.cc:648] HybridClock initialized: now 1786423026397508 us; error 50 us; skew 500 ppm
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:06.400141 12236 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:37:06.401012 12236 webserver.cc:460] Webserver started at http://127.2.238.131:42103/ using document root <none> and password file <none>
I20260811 04:37:06.401543 12236 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:06.401643 12236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:06.401937 12236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:06.403738 12236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "c459a16d54734e3e8229ead746e101d4"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "43517a58dd21a4ebf6a738f362eec6d2"
server_key_iv: "9fb23d10eb62457f01f68ddc2568d05d"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.404260 12236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "c459a16d54734e3e8229ead746e101d4"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "43517a58dd21a4ebf6a738f362eec6d2"
server_key_iv: "9fb23d10eb62457f01f68ddc2568d05d"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.407711 12236 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:37:06.409967 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.411041 12236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:06.411171 12236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "c459a16d54734e3e8229ead746e101d4"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "43517a58dd21a4ebf6a738f362eec6d2"
server_key_iv: "9fb23d10eb62457f01f68ddc2568d05d"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.411276 12236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:06.430385 12236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:06.432958 12236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:06.433163 12236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:06.433705 12236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:06.434507 12236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:06.434578 12236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.434659 12236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:06.434702 12236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.443765 12236 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:43293
I20260811 04:37:06.443800 12370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:43293 every 8 connection(s)
I20260811 04:37:06.444547 12236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:06.453176 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12236
I20260811 04:37:06.453315 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:06.453603 3002 external_mini_cluster.cc:1428] Setting key 697b5072f70b8ec1dc8d12d948c4ecf8
I20260811 04:37:06.457484 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.446357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:49247 (local address 127.2.238.190:40775)
0811 04:37:06.446491 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.446494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.447260 (+ 766us) server_negotiation.cc:407] Connection header received
0811 04:37:06.448495 (+ 1235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.448498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.448584 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.448699 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.449811 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.450272 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.451405 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.451634 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.451764 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.454160 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:06.454163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:06.454164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:06.454184 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:06.455535 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.456054 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.456059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.456103 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.456216 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:06.456529 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:06.456532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:06.456533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:06.456763 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:06.456843 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:06.457275 (+ 432us) server_negotiation.cc:299] Negotiation successful
0811 04:37:06.457371 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:37:06.458289 12373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.446655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:49247)
0811 04:37:06.447097 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.447154 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.448300 (+ 1146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.448865 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.448872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.449213 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.449643 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.449655 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.450392 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.450399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.451273 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.451284 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.451803 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.451806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.451953 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.452511 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.452525 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.454005 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:06.455670 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.455679 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.455692 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.455965 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.456321 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:06.456323 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:06.456324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:06.456444 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:06.456855 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:06.456860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:06.457069 (+ 209us) client_negotiation.cc:769] Sending connection context
0811 04:37:06.457215 (+ 146us) client_negotiation.cc:240] Negotiation successful
0811 04:37:06.457445 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":116,"threads_started":1}
I20260811 04:37:06.459408 12371 heartbeater.cc:347] Connected to a master server at 127.2.238.190:40775
I20260811 04:37:06.459628 12371 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:06.460106 12371 heartbeater.cc:511] Master 127.2.238.190:40775 requested a full tablet report, sending...
I20260811 04:37:06.461380 11891 ts_manager.cc:194] Registered new tserver with Master: c459a16d54734e3e8229ead746e101d4 (127.2.238.131:43293)
I20260811 04:37:06.462010 11891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:49247
I20260811 04:37:06.467199 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
Entry for principal oryx/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab.
WARNING: no policy specified for oryx/127.2.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
Entry for principal oryx/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab.
WARNING: no policy specified for oryx/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
Entry for principal oryx/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab.
I20260811 04:37:06.536584 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11956
I20260811 04:37:06.544793 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12092
I20260811 04:37:06.550870 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12236
I20260811 04:37:06.558079 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 11857
I20260811 04:37:06.565651 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40775
--webserver_interface=127.2.238.190
--webserver_port=41285
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:40775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:06.672101 12380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:06.672319 12380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:06.672355 12380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:06.676155 12380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:06.676201 12380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:06.676218 12380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:06.676232 12380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:06.676272 12380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:06.681295 12380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:40775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:40775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=41285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:06.682257 12380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:06.683218 12380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:06.691012 12388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.690965 12386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.690964 12385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:06.691176 12380 server_base.cc:1034] running on GCE node
I20260811 04:37:06.691838 12380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:06.692740 12380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:06.693881 12380 hybrid_clock.cc:648] HybridClock initialized: now 1786423026693849 us; error 33 us; skew 500 ppm
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:06.696575 12380 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:37:06.697410 12380 webserver.cc:460] Webserver started at http://127.2.238.190:41285/ using document root <none> and password file <none>
I20260811 04:37:06.697979 12380 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:06.698053 12380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:06.701538 12380 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:06.703701 12395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.704768 12380 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:37:06.704910 12380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "0756dd4d1be44d14bb9c997c52e87f41"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "f16095c26f43c67716b9b183214cdefb"
server_key_iv: "a8001a2c8ecb1d02f06334630a28b7d7"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.705355 12380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:06.727394 12380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:06.730242 12380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:06.730427 12380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:06.737730 12380 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:40775
I20260811 04:37:06.737763 12450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:40775 every 8 connection(s)
I20260811 04:37:06.738878 12380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:06.741389 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12380
I20260811 04:37:06.741918 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:39085
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:40775
--webserver_port=41063
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:37:06.743351 12451 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:37:06.745111 12451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Bootstrap starting.
I20260811 04:37:06.762848 12451 log.cc:824] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:06.769107 12451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:37:06.769480 12451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Bootstrap complete.
I20260811 04:37:06.773622 12451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:06.774091 12451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0756dd4d1be44d14bb9c997c52e87f41, State: Initialized, Role: FOLLOWER
I20260811 04:37:06.774607 12451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:06.774761 12451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:06.774868 12451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:06.774977 12451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:37:06.776782 12451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:06.777089 12451 leader_election.cc:302] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0756dd4d1be44d14bb9c997c52e87f41; no voters:
I20260811 04:37:06.777405 12451 leader_election.cc:288] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 04:37:06.777484 12456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 04:37:06.777681 12456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [term 2 LEADER]: Becoming Leader. State: Replica: 0756dd4d1be44d14bb9c997c52e87f41, State: Running, Role: LEADER
I20260811 04:37:06.777978 12456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } }
I20260811 04:37:06.778678 12451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:06.778877 12458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0756dd4d1be44d14bb9c997c52e87f41. Latest consensus state: current_term: 2 leader_uuid: "0756dd4d1be44d14bb9c997c52e87f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } } }
I20260811 04:37:06.778992 12458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:06.779693 12457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0756dd4d1be44d14bb9c997c52e87f41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0756dd4d1be44d14bb9c997c52e87f41" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 40775 } } }
I20260811 04:37:06.779814 12457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:06.780330 12460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:06.783371 12460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:06.783973 12460 catalog_manager.cc:1259] Loaded cluster ID: b2a560ad148847ea946f2be216920da3
I20260811 04:37:06.784034 12460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:06.786453 12460 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:06.787140 12460 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0756dd4d1be44d14bb9c997c52e87f41: Loaded TSK: 0
I20260811 04:37:06.788355 12460 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:37:06.856032 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:06.856290 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:06.856387 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:06.860173 12453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:06.860260 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:06.860342 12453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:06.865352 12453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.129.keytab
--principal=oryx/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=41063
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:06.866350 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:06.867290 12453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:06.877070 12480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.877074 12479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:06.877408 12482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:06.877830 12453 server_base.cc:1034] running on GCE node
I20260811 04:37:06.878304 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:06.878948 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:06.880118 12453 hybrid_clock.cc:648] HybridClock initialized: now 1786423026880101 us; error 27 us; skew 500 ppm
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423026, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:06.883200 12453 init.cc:377] Logged in from keytab as oryx/127.2.238.129@KRBTEST.COM (short username oryx)
I20260811 04:37:06.884235 12453 webserver.cc:460] Webserver started at http://127.2.238.129:41063/ using document root <none> and password file <none>
I20260811 04:37:06.884784 12453 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:06.884897 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:06.888819 12453 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:06.890865 12490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.892139 12453 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:06.892264 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "20d8711efe8746c8a3f0f7cfb008fd92"
format_stamp: "Formatted at 2026-08-11 04:37:05 on dist-test-slave-0kls"
server_key: "42d03b44d055c2720ef173ac388d75d0"
server_key_iv: "4b44bef940e4aec4c87312f21867b140"
server_key_version: "encryptionkey@0"
I20260811 04:37:06.892768 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:06.907711 12453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:06.911235 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:06.911442 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:06.912137 12453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:06.913060 12453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:06.913107 12453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.913147 12453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:06.913187 12453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:06.922489 12453 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:39085
I20260811 04:37:06.922523 12610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:39085 every 8 connection(s)
I20260811 04:37:06.923452 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:06.929078 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12453
I20260811 04:37:06.929533 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:46881
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:40775
--webserver_port=40831
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:06.936187 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.925519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:56227)
0811 04:37:06.926046 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.926094 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.927243 (+ 1149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.928278 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.928285 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.928614 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.929084 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.929098 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.930553 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.930557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.931421 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.931431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.932193 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.932197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.932366 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.932926 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.932940 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.935287 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":158,"threads_started":1}
W20260811 04:37:06.936178 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:06.925381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56227 (local address 127.2.238.190:40775)
0811 04:37:06.925874 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.925891 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.926212 (+ 321us) server_negotiation.cc:407] Connection header received
0811 04:37:06.927554 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.927565 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.927905 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.928186 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.929278 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.930409 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.931558 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.932029 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.932201 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.935485 (+ 3284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:56227: BlockingRecv error: recv got EOF from 127.2.238.129:56227 (error 108)
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":154,"threads_started":1}
W20260811 04:37:06.936601 12611 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.238.190:40775: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:06.942672 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.936984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:42165)
0811 04:37:06.937244 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.937255 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.937322 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.937568 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.937570 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.937689 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.937871 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.937878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.938950 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.938952 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.939465 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.939472 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.940168 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.940171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.940240 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.940731 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.940742 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.942531 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":101}
W20260811 04:37:06.942782 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:06.937062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:42165 (local address 127.2.238.190:40775)
0811 04:37:06.937224 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.937231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.937313 (+ 82us) server_negotiation.cc:407] Connection header received
0811 04:37:06.937386 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.937390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.937447 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.937546 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.937990 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.938839 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.939609 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.940046 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.940178 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.942658 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:42165: BlockingRecv error: recv got EOF from 127.2.238.129:42165 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:06 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:06.947983 12614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:06.943154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:54391)
0811 04:37:06.943273 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:06.943288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:06.943361 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:06.943575 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:06.943578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:06.943736 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:06.943822 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.943828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.944531 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.944534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.945360 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:06.945370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.945756 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.945759 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:06.945864 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.946253 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:06.946266 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:06.947900 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 04:37:06.948164 12611 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:37:06.948315 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:06.943270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:54391 (local address 127.2.238.190:40775)
0811 04:37:06.943388 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:06.943391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:06.943402 (+ 11us) server_negotiation.cc:407] Connection header received
0811 04:37:06.943432 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:06.943434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:06.943466 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:06.943529 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:06.943966 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.944432 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.945469 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:06.945658 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:06.945738 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:06.948043 (+ 2305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:54391: BlockingRecv error: recv got EOF from 127.2.238.129:54391 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260811 04:37:07.041646 12615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:07.041923 12615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:07.041967 12615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:07.045832 12615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:07.045897 12615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:07.045951 12615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:07.050872 12615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.130.keytab
--principal=oryx/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:46881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=40831
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:07.051877 12615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:07.052803 12615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:07.062323 12624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:07.062342 12621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:07.062474 12615 server_base.cc:1034] running on GCE node
W20260811 04:37:07.062511 12622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:07.062929 12615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:07.063568 12615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:07.064759 12615 hybrid_clock.cc:648] HybridClock initialized: now 1786423027064751 us; error 32 us; skew 500 ppm
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423027, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:07.067405 12615 init.cc:377] Logged in from keytab as oryx/127.2.238.130@KRBTEST.COM (short username oryx)
I20260811 04:37:07.068332 12615 webserver.cc:460] Webserver started at http://127.2.238.130:40831/ using document root <none> and password file <none>
I20260811 04:37:07.068868 12615 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:07.068980 12615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:07.072839 12615 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:07.074946 12631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.076164 12615 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:07.076295 12615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "f260c4639aab441e87cc5bab4ae6702f"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "663158af3509712a4e20639da519c198"
server_key_iv: "2013e7227c1941915e37e411e3ccb905"
server_key_version: "encryptionkey@0"
I20260811 04:37:07.076810 12615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:07.088238 12615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:07.090819 12615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:07.091005 12615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:07.091545 12615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:07.092370 12615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:07.092440 12615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.092509 12615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:07.092550 12615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.101644 12615 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:46881
I20260811 04:37:07.101677 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:46881 every 8 connection(s)
I20260811 04:37:07.102658 12615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:07.106637 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12615
I20260811 04:37:07.107187 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:43293
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:40775
--webserver_port=42103
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260811 04:37:07.114816 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.104421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:51361 (local address 127.2.238.190:40775)
0811 04:37:07.104546 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.104550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.105322 (+ 772us) server_negotiation.cc:407] Connection header received
0811 04:37:07.106826 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.106829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.106891 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.106961 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.108477 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.109088 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.110741 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.111152 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.111283 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.114720 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:51361: BlockingRecv error: recv got EOF from 127.2.238.130:51361 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:07.115588 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.104786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:51361)
0811 04:37:07.105190 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.105227 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.106549 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.107180 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.107189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.107660 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.108326 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.108346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.109218 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.109224 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.110597 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.110614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.111294 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.111297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.111434 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.112032 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.112047 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.114595 (+ 2548us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":111,"threads_started":1}
W20260811 04:37:07.116040 12752 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.238.190:40775: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:07.123291 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.117277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:41163)
0811 04:37:07.117824 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.117842 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.117945 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.118274 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.118297 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.118477 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.118615 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.118624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.119431 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.119437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.120453 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.120465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.120891 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.120895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.120991 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.121395 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.121409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.123185 (+ 1776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421}
W20260811 04:37:07.123551 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.117399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41163 (local address 127.2.238.190:40775)
0811 04:37:07.117494 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.117497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.117929 (+ 432us) server_negotiation.cc:407] Connection header received
0811 04:37:07.118044 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.118047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.118121 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.118205 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.118757 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.119317 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.120580 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.120786 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.120863 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.123297 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:41163: BlockingRecv error: recv got EOF from 127.2.238.130:41163 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:07.128809 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.123747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:35195)
0811 04:37:07.123843 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.123857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.123925 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.124249 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.124254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.124397 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.124474 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.124480 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.125071 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.125074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.125856 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.125867 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.126256 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.126260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.126334 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.126692 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.126710 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.128696 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260811 04:37:07.129134 12752 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:37:07.129204 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.123985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:35195 (local address 127.2.238.190:40775)
0811 04:37:07.124078 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.124080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.124093 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:07.124119 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.124120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.124149 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.124201 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.124575 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.124982 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.125959 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.126157 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.126231 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.129109 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:35195: BlockingRecv error: recv got EOF from 127.2.238.130:35195 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260811 04:37:07.238596 12755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:07.238871 12755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:07.238983 12755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:07.242967 12755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:07.243045 12755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:07.243132 12755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:07.248134 12755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/krb5kdc/oryx_127.2.238.131.keytab
--principal=oryx/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:43293
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=42103
--tserver_master_addrs=127.2.238.190:40775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:07.249202 12755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:07.250422 12755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:07.260430 12763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:07.260430 12761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:07.260449 12760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:07.260996 12755 server_base.cc:1034] running on GCE node
I20260811 04:37:07.261446 12755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:07.262210 12755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:07.263422 12755 hybrid_clock.cc:648] HybridClock initialized: now 1786423027263386 us; error 44 us; skew 500 ppm
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423027, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:07.266520 12755 init.cc:377] Logged in from keytab as oryx/127.2.238.131@KRBTEST.COM (short username oryx)
I20260811 04:37:07.267450 12755 webserver.cc:460] Webserver started at http://127.2.238.131:42103/ using document root <none> and password file <none>
I20260811 04:37:07.268046 12755 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:07.268168 12755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:07.272328 12755 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:37:07.274614 12771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.275902 12755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:37:07.276057 12755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "c459a16d54734e3e8229ead746e101d4"
format_stamp: "Formatted at 2026-08-11 04:37:06 on dist-test-slave-0kls"
server_key: "43517a58dd21a4ebf6a738f362eec6d2"
server_key_iv: "9fb23d10eb62457f01f68ddc2568d05d"
server_key_version: "encryptionkey@0"
I20260811 04:37:07.276599 12755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:07.299882 12755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:07.303392 12755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:07.303607 12755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:07.304353 12755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:07.305384 12755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:07.305461 12755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.305585 12755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:07.305637 12755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:07.315083 12755 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:43293
I20260811 04:37:07.315092 12892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:43293 every 8 connection(s)
I20260811 04:37:07.316109 12755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:07.325454 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12755
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
W20260811 04:37:07.327798 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.318042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53751 (local address 127.2.238.190:40775)
0811 04:37:07.318199 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.318202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.319092 (+ 890us) server_negotiation.cc:407] Connection header received
0811 04:37:07.320483 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.320487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.320549 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.320634 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.321847 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.322412 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.323543 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.323770 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.323891 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.327682 (+ 3791us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:53751: BlockingRecv error: recv got EOF from 127.2.238.131:53751 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260811 04:37:07.328433 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.318476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:53751)
0811 04:37:07.318921 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.318964 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.320263 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.320811 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.320818 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.321174 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.321666 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.321681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.322576 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.322579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.323417 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.323426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.323933 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.323937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.324098 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.324696 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.324712 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.327573 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":120,"threads_started":1}
W20260811 04:37:07.328959 12893 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:40775 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.238.190:40775: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:07.334465 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.329381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:47313)
0811 04:37:07.329558 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.329570 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.329643 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.329969 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.329972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.330124 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.330215 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.330220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.330949 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.330954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.331487 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.331496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.331976 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.331979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.332071 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.332590 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.332600 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.334340 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115}
W20260811 04:37:07.334542 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.329491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:47313 (local address 127.2.238.190:40775)
0811 04:37:07.329595 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.329601 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.329615 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:07.329759 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.329763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.329866 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.329952 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.330348 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.330837 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.331617 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.331864 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.331960 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.334438 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:47313: BlockingRecv error: recv got EOF from 127.2.238.131:47313 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 04:37:07.335808 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.326560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.2.238.190:40775)
0811 04:37:07.326825 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.326829 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.326841 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:37:07.326885 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.326887 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.326920 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.326991 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.328042 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.328611 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.329548 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.329762 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.329912 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.331290 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:07.331295 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:07.331310 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:07.331336 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:07.333287 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:07.333750 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:07.333757 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:07.333777 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:07.333952 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:07.334240 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:07.334243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:07.334245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:07.334643 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:07.334851 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:07.335132 (+ 281us) server_negotiation.cc:299] Negotiation successful
0811 04:37:07.335616 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:07.340211 12895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.335181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:43475)
0811 04:37:07.335333 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.335347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.335415 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.335616 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.335618 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.335764 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.335855 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.335861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.336520 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.336523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.337069 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.337078 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.337860 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.337868 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.337943 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.338423 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.338437 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.340125 (+ 1688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260811 04:37:07.340389 12613 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.335246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:43475 (local address 127.2.238.190:40775)
0811 04:37:07.335366 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.335369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.335379 (+ 10us) server_negotiation.cc:407] Connection header received
0811 04:37:07.335487 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.335489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.335517 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.335580 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.335977 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.336397 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.337175 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.337463 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.337590 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.340239 (+ 2649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:43475: BlockingRecv error: recv got EOF from 127.2.238.131:43475 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260811 04:37:07.340386 12893 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:07 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:07.954689 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:07.948917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:53311)
0811 04:37:07.949202 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:07.949215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:07.949315 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:07.949550 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:07.949552 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:07.949724 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:07.949866 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.949890 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.950723 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.950726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.951525 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:07.951536 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.952020 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.952024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:07.952130 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.952696 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:07.952707 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:07.954578 (+ 1871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":116,"threads_started":1}
W20260811 04:37:07.954835 12899 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:07.949016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:53311 (local address 127.2.238.190:40775)
0811 04:37:07.949261 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:07.949264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:07.949278 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:07.949396 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:07.949398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:07.949442 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:07.949515 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:07.949985 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.950542 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.951652 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:07.951901 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:07.952014 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:07.954704 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:53311: BlockingRecv error: recv got EOF from 127.2.238.129:53311 (error 108)
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:08.135262 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:08.129768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:45335)
0811 04:37:08.130045 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:08.130058 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:08.130181 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:08.130444 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:08.130447 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:08.130640 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:08.130767 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.130773 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.131543 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.131548 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.132107 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.132115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.132545 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.132548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.132638 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.133163 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:08.133174 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:08.135119 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":78,"threads_started":1}
W20260811 04:37:08.135329 12899 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:08.129853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45335 (local address 127.2.238.190:40775)
0811 04:37:08.130011 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:08.130016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:08.130133 (+ 117us) server_negotiation.cc:407] Connection header received
0811 04:37:08.130301 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:08.130303 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:08.130345 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:08.130423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:08.130894 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.131427 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.132235 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.132440 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.132539 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.135228 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:45335: BlockingRecv error: recv got EOF from 127.2.238.130:45335 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:08.346371 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:08.340971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:37013)
0811 04:37:08.341244 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:08.341259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:08.341380 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:08.341682 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:08.341685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:08.341922 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:08.342018 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.342024 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.342804 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.342809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.343418 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.343426 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.343905 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.343909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.344005 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.344483 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:08.344494 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:08.346256 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":83,"threads_started":1}
W20260811 04:37:08.346540 12899 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:08.341060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:37013 (local address 127.2.238.190:40775)
0811 04:37:08.341197 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:08.341203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:08.341356 (+ 153us) server_negotiation.cc:407] Connection header received
0811 04:37:08.341497 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:08.341501 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:08.341551 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:08.341654 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:08.342145 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.342690 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.343542 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.343785 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.343889 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.346386 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:37013: BlockingRecv error: recv got EOF from 127.2.238.131:37013 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:08 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:08.960392 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:08.955586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:35153)
0811 04:37:08.955874 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:08.955888 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:08.955990 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:08.956227 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:08.956228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:08.956398 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:08.956489 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.956494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.957193 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.957195 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.957703 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:08.957710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.958087 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.958090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:08.958147 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.958597 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:08.958607 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:08.960213 (+ 1606us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":69,"threads_started":1}
W20260811 04:37:08.960528 12903 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:08.955654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:35153 (local address 127.2.238.190:40775)
0811 04:37:08.955916 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:08.955919 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:08.955946 (+ 27us) server_negotiation.cc:407] Connection header received
0811 04:37:08.956068 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:08.956070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:08.956110 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:08.956177 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:08.956596 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.957082 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.957807 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:08.957995 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:08.958061 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:08.960367 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:35153: BlockingRecv error: recv got EOF from 127.2.238.129:35153 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":128,"threads_started":1}
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:09.140972 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:09.136075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:41491)
0811 04:37:09.136320 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:09.136334 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:09.136435 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:09.136669 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:09.136671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:09.136843 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:09.136940 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.136945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.137653 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.137656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.138175 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.138200 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.138616 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.138618 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.138675 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.139125 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:09.139136 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:09.140857 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
W20260811 04:37:09.141120 12903 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:09.136130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41491 (local address 127.2.238.190:40775)
0811 04:37:09.136257 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:09.136261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:09.136403 (+ 142us) server_negotiation.cc:407] Connection header received
0811 04:37:09.136509 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:09.136511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:09.136553 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:09.136614 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:09.137031 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.137554 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.138295 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.138523 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.138589 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.140999 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:41491: BlockingRecv error: recv got EOF from 127.2.238.130:41491 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:09.352479 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:09.347226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:59935)
0811 04:37:09.347546 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:09.347560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:09.347660 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:09.347880 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:09.347883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:09.348067 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:09.348177 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.348182 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.348831 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.348833 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.349328 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.349334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.349718 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.349720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.349836 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.350404 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:09.350416 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:09.352340 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":139,"threads_started":1}
W20260811 04:37:09.352609 12903 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:09.347367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:59935 (local address 127.2.238.190:40775)
0811 04:37:09.347519 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:09.347523 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:09.347631 (+ 108us) server_negotiation.cc:407] Connection header received
0811 04:37:09.347725 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:09.347727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:09.347773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:09.347840 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:09.348273 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.348743 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.349422 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.349621 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.349696 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.352472 (+ 2776us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:59935: BlockingRecv error: recv got EOF from 127.2.238.131:59935 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:09 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:09.966848 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:09.961430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:50895)
0811 04:37:09.961776 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:09.961833 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:09.961946 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:09.962178 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:09.962180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:09.962358 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:09.962460 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.962465 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.963217 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.963220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.963793 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:09.963801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.964226 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.964229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:09.964287 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.964786 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:09.964798 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:09.966722 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
W20260811 04:37:09.967027 12907 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:09.961514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:50895 (local address 127.2.238.190:40775)
0811 04:37:09.961832 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:09.961835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:09.961893 (+ 58us) server_negotiation.cc:407] Connection header received
0811 04:37:09.962014 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:09.962016 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:09.962058 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:09.962132 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:09.962566 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.963096 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.963903 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:09.964128 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:09.964202 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:09.966882 (+ 2680us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:50895: BlockingRecv error: recv got EOF from 127.2.238.129:50895 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":117,"threads_started":1}
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:10.147704 12908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:10.141924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:48931)
0811 04:37:10.142227 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:10.142242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:10.142351 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:10.142683 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:10.142689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:10.142896 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:10.143076 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.143088 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.143984 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.143987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.144569 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.144578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.145010 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.145013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.145103 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.145580 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:10.145593 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:10.147581 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":91,"threads_started":1}
W20260811 04:37:10.147872 12907 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:10.142082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:48931 (local address 127.2.238.190:40775)
0811 04:37:10.142244 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:10.142248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:10.142299 (+ 51us) server_negotiation.cc:407] Connection header received
0811 04:37:10.142455 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:10.142458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:10.142516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:10.142607 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:10.143221 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.143877 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.144677 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.144907 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.144984 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.147718 (+ 2734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:48931: BlockingRecv error: recv got EOF from 127.2.238.130:48931 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:10.358659 12909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:10.353390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:56809)
0811 04:37:10.353692 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:10.353707 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:10.353855 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:10.354084 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:10.354086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:10.354262 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:10.354361 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.354366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.355054 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.355057 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.355707 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.355716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.356108 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.356111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.356171 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.356632 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:10.356643 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:10.358532 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":132,"threads_started":1}
W20260811 04:37:10.358808 12907 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:10.353467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:56809 (local address 127.2.238.190:40775)
0811 04:37:10.353636 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:10.353640 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:10.353805 (+ 165us) server_negotiation.cc:407] Connection header received
0811 04:37:10.353948 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:10.353950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:10.353997 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:10.354070 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:10.354459 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.354962 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.355812 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.356015 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.356087 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.358677 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:56809: BlockingRecv error: recv got EOF from 127.2.238.131:56809 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:10 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:10.973320 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:10.967710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:44701)
0811 04:37:10.968071 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:10.968085 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:10.968197 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:10.968450 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:10.968452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:10.968623 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:10.968720 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.968725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.969467 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.969469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.970060 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:10.970068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.970495 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.970497 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:10.970556 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.971184 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:10.971200 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:10.973201 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":121,"threads_started":1}
W20260811 04:37:10.973459 12911 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:10.967889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:44701 (local address 127.2.238.190:40775)
0811 04:37:10.968119 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:10.968122 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:10.968152 (+ 30us) server_negotiation.cc:407] Connection header received
0811 04:37:10.968286 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:10.968288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:10.968331 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:10.968406 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:10.968818 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.969369 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.970174 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:10.970387 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:10.970468 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:10.973325 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:44701: BlockingRecv error: recv got EOF from 127.2.238.129:44701 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":61,"threads_started":1}
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:11.154093 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:11.148493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:57843)
0811 04:37:11.148872 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:11.148886 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:11.148984 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:11.149217 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:11.149219 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:11.149397 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:11.149493 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.149498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.150281 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.150284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.150822 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.150830 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.151254 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.151257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.151321 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.151801 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:11.151813 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:11.153935 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":70,"threads_started":1}
W20260811 04:37:11.154248 12911 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:11.148591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:57843 (local address 127.2.238.190:40775)
0811 04:37:11.148795 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:11.148799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:11.148958 (+ 159us) server_negotiation.cc:407] Connection header received
0811 04:37:11.149065 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:11.149067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:11.149111 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:11.149178 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:11.149592 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.150183 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.150945 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.151152 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.151226 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.154095 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:57843: BlockingRecv error: recv got EOF from 127.2.238.130:57843 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:11.369517 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:11.359475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:45993)
0811 04:37:11.359767 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:11.359780 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:11.359891 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:11.360143 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:11.360146 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:11.360332 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:11.360432 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.360437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.361176 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.361179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.361740 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.361748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.362385 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.362390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.362495 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.367436 (+ 4941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:11.367450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:11.369336 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":90,"threads_started":1}
W20260811 04:37:11.369571 12911 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:11.359537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:45993 (local address 127.2.238.190:40775)
0811 04:37:11.359689 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:11.359693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:11.359865 (+ 172us) server_negotiation.cc:407] Connection header received
0811 04:37:11.359981 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:11.359983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:11.360030 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:11.360100 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:11.360534 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.361067 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.361887 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.362242 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.362365 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.369464 (+ 7099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:45993: BlockingRecv error: recv got EOF from 127.2.238.131:45993 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:11 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:11.979918 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:11.974314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:38599)
0811 04:37:11.974631 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:11.974645 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:11.974734 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:11.974987 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:11.974989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:11.975178 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:11.975319 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.975330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.976220 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.976223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.976818 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:11.976827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.977248 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.977250 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:11.977307 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.977772 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:11.977812 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:11.979742 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":89,"threads_started":1}
W20260811 04:37:11.980103 12915 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:11.974455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:38599 (local address 127.2.238.190:40775)
0811 04:37:11.974721 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:11.974725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:11.974739 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:11.974827 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:11.974840 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:11.974880 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:11.974949 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:11.975441 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.976107 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.976920 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:11.977153 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:11.977227 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:11.979894 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:38599: BlockingRecv error: recv got EOF from 127.2.238.129:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:12.160717 12916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:12.154938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:53825)
0811 04:37:12.155241 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:12.155258 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:12.155392 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:12.155682 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:12.155685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:12.155944 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:12.156046 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.156052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.156843 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.156849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.157434 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.157442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.157980 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.157984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.158068 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.158536 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:12.158547 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:12.160601 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
W20260811 04:37:12.160988 12915 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:12.155129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53825 (local address 127.2.238.190:40775)
0811 04:37:12.155261 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:12.155265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:12.155322 (+ 57us) server_negotiation.cc:407] Connection header received
0811 04:37:12.155510 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:12.155514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:12.155570 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:12.155661 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:12.156178 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.156723 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.157559 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.157840 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.157973 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.160745 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:53825: BlockingRecv error: recv got EOF from 127.2.238.130:53825 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:12.376030 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:12.370499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:52459)
0811 04:37:12.370767 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:12.370793 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:12.370897 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:12.371140 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:12.371143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:12.371330 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:12.371432 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.371437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.372165 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.372167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.372705 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.372712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.373123 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.373125 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.373183 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.373761 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:12.373776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:12.375917 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":86,"threads_started":1}
W20260811 04:37:12.376222 12915 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:12.370561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:52459 (local address 127.2.238.190:40775)
0811 04:37:12.370711 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:12.370716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:12.370872 (+ 156us) server_negotiation.cc:407] Connection header received
0811 04:37:12.370981 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:12.370983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:12.371033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:12.371122 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:12.371533 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.372049 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.372803 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.373028 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.373118 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.376050 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:52459: BlockingRecv error: recv got EOF from 127.2.238.131:52459 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:12 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:12.986670 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:12.980740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:41235)
0811 04:37:12.981121 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:12.981141 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:12.981264 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:12.981612 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:12.981615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:12.981880 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:12.981981 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.981987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.982772 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.982777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.983359 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:12.983367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.983863 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.983867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:12.983958 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.984464 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:12.984476 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:12.986520 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":94,"threads_started":1}
W20260811 04:37:12.986976 12919 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:12.980817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:41235 (local address 127.2.238.190:40775)
0811 04:37:12.981121 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:12.981128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:12.981202 (+ 74us) server_negotiation.cc:407] Connection header received
0811 04:37:12.981400 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:12.981403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:12.981480 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:12.981593 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:12.982109 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.982660 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.983483 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:12.983738 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:12.983847 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:12.986680 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:41235: BlockingRecv error: recv got EOF from 127.2.238.129:41235 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:13.167176 12920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:13.161519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:45349)
0811 04:37:13.161854 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:13.161867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:13.161966 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:13.162243 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:13.162244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:13.162450 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:13.162544 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.162548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.163352 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.163354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.163903 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.163911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.164397 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.164399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.164459 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.165061 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:13.165074 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:13.167058 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
W20260811 04:37:13.167395 12919 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:13.161607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45349 (local address 127.2.238.190:40775)
0811 04:37:13.161756 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:13.161760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:13.161937 (+ 177us) server_negotiation.cc:407] Connection header received
0811 04:37:13.162033 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:13.162035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:13.162109 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:13.162191 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:13.162641 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.163230 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.164032 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.164286 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.164376 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.167274 (+ 2898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:45349: BlockingRecv error: recv got EOF from 127.2.238.130:45349 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:13.383327 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:13.377004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:58087)
0811 04:37:13.377388 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:13.377405 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:13.377524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:13.377840 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:13.377843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:13.378071 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:13.378194 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.378199 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.379140 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.379146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.379936 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.379948 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.380537 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.380540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.380616 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.381271 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:13.381297 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:13.383201 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":123,"threads_started":1}
W20260811 04:37:13.383460 12919 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:13.377228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:58087 (local address 127.2.238.190:40775)
0811 04:37:13.377388 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:13.377394 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:13.377465 (+ 71us) server_negotiation.cc:407] Connection header received
0811 04:37:13.377638 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:13.377640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:13.377701 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:13.377785 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:13.378314 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.379030 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.380096 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.380417 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.380535 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.383340 (+ 2805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:58087: BlockingRecv error: recv got EOF from 127.2.238.131:58087 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:13 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:13.992785 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:13.987443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:46363)
0811 04:37:13.987820 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:13.987834 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:13.987927 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:13.988214 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:13.988216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:13.988431 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:13.988530 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.988535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.989279 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.989281 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.989908 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:13.989918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.990307 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.990309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:13.990364 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.990886 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:13.990898 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:13.992655 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":103,"threads_started":1}
W20260811 04:37:13.992913 12923 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:13.987651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:46363 (local address 127.2.238.190:40775)
0811 04:37:13.987870 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:13.987873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:13.987886 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:13.988009 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:13.988011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:13.988071 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:13.988176 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:13.988643 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.989183 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.990003 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:13.990217 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:13.990286 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:13.992789 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:46363: BlockingRecv error: recv got EOF from 127.2.238.129:46363 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:14.173410 12924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:14.168036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:53285)
0811 04:37:14.168378 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:14.168393 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:14.168503 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:14.168738 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:14.168740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:14.168933 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:14.169031 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.169037 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.169756 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.169758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.170374 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.170384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.170805 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.170807 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.170866 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.171466 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:14.171479 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:14.173301 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
W20260811 04:37:14.173555 12923 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:14.168108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53285 (local address 127.2.238.190:40775)
0811 04:37:14.168262 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:14.168265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:14.168472 (+ 207us) server_negotiation.cc:407] Connection header received
0811 04:37:14.168581 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:14.168583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:14.168630 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:14.168698 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:14.169142 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.169664 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.170492 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.170700 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.170771 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.173436 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:53285: BlockingRecv error: recv got EOF from 127.2.238.130:53285 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:14.390516 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:14.384356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:46565)
0811 04:37:14.384712 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:14.384725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:14.384876 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:14.385160 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:14.385163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:14.385388 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:14.385490 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.385496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.386456 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.386459 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.387380 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.387394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.387880 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.387882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.387948 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.388451 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:14.388462 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:14.390401 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":87,"threads_started":1}
W20260811 04:37:14.390764 12923 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:14.384423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:46565 (local address 127.2.238.190:40775)
0811 04:37:14.384595 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:14.384599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:14.384814 (+ 215us) server_negotiation.cc:407] Connection header received
0811 04:37:14.384980 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:14.384983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:14.385041 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:14.385129 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:14.385621 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.386308 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.387510 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.387761 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.387851 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.390544 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:46565: BlockingRecv error: recv got EOF from 127.2.238.131:46565 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:14 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:14.999532 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:14.993726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:40277)
0811 04:37:14.994078 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:14.994095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:14.994200 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:14.994503 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:14.994505 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:14.994756 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:14.994901 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.994910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.995823 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.995828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.996484 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:14.996493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.996977 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.996981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:14.997066 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.997589 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:14.997601 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:14.999359 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
W20260811 04:37:14.999605 12927 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:14.993929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:40277 (local address 127.2.238.190:40775)
0811 04:37:14.994172 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:14.994176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:14.994190 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:14.994293 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:14.994296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:14.994339 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:14.994441 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:14.995032 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.995698 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.996634 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:14.996864 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:14.996970 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:14.999487 (+ 2517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:40277: BlockingRecv error: recv got EOF from 127.2.238.129:40277 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":92,"threads_started":1}
Aug 11 04:37:15 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:15 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:15.180140 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:15.174363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:58117)
0811 04:37:15.174635 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:15.174649 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:15.174750 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:15.175063 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:15.175065 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:15.175290 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:15.175395 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:15.175401 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.176281 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.176283 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:15.176844 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:15.176852 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.177311 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.177314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:15.177385 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:15.177953 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:15.177966 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:15.179959 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":97,"threads_started":1}
W20260811 04:37:15.180337 12927 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:15.174445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:58117 (local address 127.2.238.190:40775)
0811 04:37:15.174593 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:15.174596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:15.174719 (+ 123us) server_negotiation.cc:407] Connection header received
0811 04:37:15.174883 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:15.174886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:15.174938 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:15.175016 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:15.175505 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.176184 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.176947 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.177195 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.177273 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:15.180157 (+ 2884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:58117: BlockingRecv error: recv got EOF from 127.2.238.130:58117 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 11 04:37:15 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:15 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:15.396859 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:15.391286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:33709)
0811 04:37:15.391597 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:15.391613 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:15.391724 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:15.391977 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:15.391979 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:15.392184 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:15.392297 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:15.392315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.393053 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.393056 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:15.393615 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:15.393624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.394101 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.394104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:15.394170 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:15.394657 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:15.394669 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:15.396735 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":112,"threads_started":1}
W20260811 04:37:15.397217 12927 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:15.391369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:33709 (local address 127.2.238.190:40775)
0811 04:37:15.391525 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:15.391530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:15.391688 (+ 158us) server_negotiation.cc:407] Connection header received
0811 04:37:15.391809 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:15.391812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:15.391861 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:15.391929 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:15.392426 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.392940 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.393722 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:15.393992 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:15.394071 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:15.396966 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:33709: BlockingRecv error: recv got EOF from 127.2.238.131:33709 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:16.008087 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:16.000545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:52563)
0811 04:37:16.000858 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:16.000871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:16.000971 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:16.001319 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:16.001321 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:16.001507 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:16.001668 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.001685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.002986 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.002989 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.003817 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.003827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.004505 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.004509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.004603 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.005332 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:16.005352 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:16.007909 (+ 2557us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
W20260811 04:37:16.008217 12931 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:16.000591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:52563 (local address 127.2.238.190:40775)
0811 04:37:16.000870 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:16.000873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:16.000923 (+ 50us) server_negotiation.cc:407] Connection header received
0811 04:37:16.001116 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:16.001119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:16.001177 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:16.001256 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:16.001851 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.002832 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.003936 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.004356 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.004490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.008085 (+ 3595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:52563: BlockingRecv error: recv got EOF from 127.2.238.129:52563 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":72,"threads_started":1}
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:16.186730 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:16.180965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:51803)
0811 04:37:16.181238 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:16.181252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:16.181353 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:16.181727 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:16.181730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:16.181989 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:16.182092 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.182098 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.182858 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.182860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.183402 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.183410 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.183853 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.183855 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.183912 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.184409 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:16.184421 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:16.186569 (+ 2148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":98,"threads_started":1}
W20260811 04:37:16.186872 12931 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:16.181056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:51803 (local address 127.2.238.190:40775)
0811 04:37:16.181228 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:16.181231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:16.181320 (+ 89us) server_negotiation.cc:407] Connection header received
0811 04:37:16.181480 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:16.181483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:16.181562 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:16.181651 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:16.182213 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.182756 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.183512 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.183745 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.183834 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.186736 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:51803: BlockingRecv error: recv got EOF from 127.2.238.130:51803 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:16 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:16.408238 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:16.397645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:46443)
0811 04:37:16.397943 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:16.397959 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:16.398104 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:16.398417 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:16.398420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:16.398629 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:16.398721 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.398726 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.399444 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.399448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.400005 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:16.400013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.400483 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.400486 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:16.400577 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.405850 (+ 5273us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:16.405863 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:16.408099 (+ 2236us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":91,"threads_started":1}
W20260811 04:37:16.408354 12931 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:16.397711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:46443 (local address 127.2.238.190:40775)
0811 04:37:16.397934 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:16.397938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:16.398049 (+ 111us) server_negotiation.cc:407] Connection header received
0811 04:37:16.398207 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:16.398210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:16.398264 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:16.398390 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:16.398843 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.399336 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.400131 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:16.400364 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:16.400476 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:16.408217 (+ 7741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:46443: BlockingRecv error: recv got EOF from 127.2.238.131:46443 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:17.014736 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:17.009104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:52981)
0811 04:37:17.009429 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:17.009442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:17.009571 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:17.009848 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:17.009851 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:17.010032 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:17.010131 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.010136 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.010877 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.010880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.011446 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.011453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.011903 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.011906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.011968 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.012442 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:17.012454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:17.014621 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
W20260811 04:37:17.014954 12936 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:17.009162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:52981 (local address 127.2.238.190:40775)
0811 04:37:17.009441 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:17.009444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:17.009554 (+ 110us) server_negotiation.cc:407] Connection header received
0811 04:37:17.009680 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:17.009682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:17.009730 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:17.009832 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:17.010238 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.010772 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.011567 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.011799 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.011881 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.014791 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:52981: BlockingRecv error: recv got EOF from 127.2.238.129:52981 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":64,"threads_started":1}
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:17.195290 12938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:17.187800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:36215)
0811 04:37:17.188261 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:17.188275 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:17.188441 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:17.188768 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:17.188770 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:17.189014 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:17.189129 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.189135 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.190320 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.190323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.191114 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.191129 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.191684 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.191691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.191803 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.192427 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:17.192459 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:17.195115 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":75,"threads_started":1}
W20260811 04:37:17.195405 12936 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:17.187837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:36215 (local address 127.2.238.190:40775)
0811 04:37:17.187984 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:17.187989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:17.188394 (+ 405us) server_negotiation.cc:407] Connection header received
0811 04:37:17.188548 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:17.188552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:17.188622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:17.188738 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:17.189263 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.190186 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.191257 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.191544 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.191632 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.195274 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:36215: BlockingRecv error: recv got EOF from 127.2.238.130:36215 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:17 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:17.414474 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:17.409042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:38741)
0811 04:37:17.409344 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:17.409357 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:17.409449 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:17.409717 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:17.409719 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:17.409964 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:17.410066 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.410071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.410749 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.410751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.411327 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:17.411353 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.411776 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.411779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:17.411841 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.412327 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:17.412338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:17.414302 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":109,"threads_started":1}
W20260811 04:37:17.414527 12936 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:17.409178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:38741 (local address 127.2.238.190:40775)
0811 04:37:17.409317 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:17.409320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:17.409421 (+ 101us) server_negotiation.cc:407] Connection header received
0811 04:37:17.409522 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:17.409524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:17.409570 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:17.409650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:17.410166 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.410661 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.411466 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:17.411678 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:17.411753 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:17.414421 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:38741: BlockingRecv error: recv got EOF from 127.2.238.131:38741 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:18.020678 12940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:18.015547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:39303)
0811 04:37:18.015835 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:18.015856 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:18.015964 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:18.016201 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:18.016203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:18.016371 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:18.016468 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.016473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.017155 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.017157 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.017664 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.017671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.018178 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.018181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.018249 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.018713 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:18.018725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:18.020543 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
W20260811 04:37:18.020928 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:18.015609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:39303 (local address 127.2.238.190:40775)
0811 04:37:18.015917 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:18.015920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:18.015933 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:18.016044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:18.016046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:18.016089 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:18.016160 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:18.016559 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.017062 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.017755 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.018051 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.018159 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.020683 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:39303: BlockingRecv error: recv got EOF from 127.2.238.129:39303 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":115,"threads_started":1}
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:18.202093 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:18.196325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:36995)
0811 04:37:18.196629 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:18.196643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:18.196748 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:18.197001 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:18.197005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:18.197219 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:18.197323 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.197328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.198329 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.198333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.199204 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.199214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.199642 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.199645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.199705 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.200200 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:18.200212 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:18.201962 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
W20260811 04:37:18.202345 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:18.196393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:36995 (local address 127.2.238.190:40775)
0811 04:37:18.196556 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:18.196560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:18.196723 (+ 163us) server_negotiation.cc:407] Connection header received
0811 04:37:18.196827 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:18.196830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:18.196877 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:18.196974 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:18.197447 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.198206 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.199309 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.199534 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.199618 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.202184 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:36995: BlockingRecv error: recv got EOF from 127.2.238.130:36995 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:18 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:18.421052 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:18.415346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:33105)
0811 04:37:18.415663 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:18.415676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:18.415801 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:18.416148 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:18.416153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:18.416408 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:18.416567 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.416576 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.417537 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.417540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.418132 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:18.418141 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.418554 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.418557 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:18.418628 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.419095 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:18.419106 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:18.420939 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":119,"threads_started":1}
W20260811 04:37:18.421204 12941 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:18.415433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:33105 (local address 127.2.238.190:40775)
0811 04:37:18.415563 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:18.415567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:18.415743 (+ 176us) server_negotiation.cc:407] Connection header received
0811 04:37:18.415897 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:18.415899 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:18.415948 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:18.416068 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:18.416701 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.417434 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.418246 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:18.418455 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:18.418533 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:18.421071 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:33105: BlockingRecv error: recv got EOF from 127.2.238.131:33105 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:19.027998 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:19.021728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:44003)
0811 04:37:19.022121 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:19.022138 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:19.022265 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:19.022542 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:19.022547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:19.022805 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:19.022959 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.022970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.023877 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.023880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.024744 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.024755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.025320 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.025323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.025382 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.025875 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:19.025887 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:19.027823 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
W20260811 04:37:19.028079 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:19.021814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:44003 (local address 127.2.238.190:40775)
0811 04:37:19.022101 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:19.022105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:19.022221 (+ 116us) server_negotiation.cc:407] Connection header received
0811 04:37:19.022353 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:19.022355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:19.022403 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:19.022481 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:19.023083 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.023765 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.024876 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.025229 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.025314 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.027947 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:44003: BlockingRecv error: recv got EOF from 127.2.238.129:44003 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":66,"threads_started":1}
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:19.207934 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:19.202930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:39673)
0811 04:37:19.203231 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:19.203244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:19.203338 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:19.203607 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:19.203609 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:19.203785 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:19.203926 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.203932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.204618 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.204620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.205122 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.205129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.205500 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.205503 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.205559 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.206025 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:19.206036 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:19.207809 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":119,"threads_started":1}
W20260811 04:37:19.208196 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:19.202966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:39673 (local address 127.2.238.190:40775)
0811 04:37:19.203120 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:19.203125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:19.203316 (+ 191us) server_negotiation.cc:407] Connection header received
0811 04:37:19.203447 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:19.203449 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:19.203496 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:19.203561 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:19.204026 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.204530 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.205216 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.205414 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.205482 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.207951 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:39673: BlockingRecv error: recv got EOF from 127.2.238.130:39673 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:19 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:19.427538 12947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:19.422018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:60785)
0811 04:37:19.422407 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:19.422419 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:19.422564 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:19.422868 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:19.422870 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:19.423074 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:19.423169 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.423175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.423922 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.423926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.424447 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:19.424465 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.424903 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.424907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:19.424991 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.425444 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:19.425455 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:19.427367 (+ 1912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":119,"threads_started":1}
W20260811 04:37:19.427600 12945 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:19.422114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:60785 (local address 127.2.238.190:40775)
0811 04:37:19.422266 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:19.422273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:19.422502 (+ 229us) server_negotiation.cc:407] Connection header received
0811 04:37:19.422686 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:19.422689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:19.422745 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:19.422843 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:19.423297 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.423803 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.424575 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:19.424789 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:19.424888 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:19.427490 (+ 2602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:60785: BlockingRecv error: recv got EOF from 127.2.238.131:60785 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:20.034122 12949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:20.028868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:42191)
0811 04:37:20.029183 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:20.029196 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:20.029297 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:20.029542 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:20.029544 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:20.029726 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:20.029887 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.029894 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.030623 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.030625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.031157 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.031164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.031605 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.031607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.031668 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.032184 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:20.032196 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:20.034003 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":108,"threads_started":1}
W20260811 04:37:20.034350 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:20.028929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:42191 (local address 127.2.238.190:40775)
0811 04:37:20.029222 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:20.029225 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:20.029241 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:37:20.029381 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:20.029383 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:20.029428 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:20.029502 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:20.029990 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.030527 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.031260 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.031504 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.031585 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.034194 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:42191: BlockingRecv error: recv got EOF from 127.2.238.129:42191 (error 108)
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":66,"threads_started":1}
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:20.214381 12951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:20.208711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:52511)
0811 04:37:20.209042 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:20.209055 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:20.209155 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:20.209380 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:20.209382 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:20.209621 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:20.209720 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.209726 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.210627 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.210631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.211414 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.211425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.211907 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.211910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.211971 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.212503 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:20.212523 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:20.214272 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":83,"threads_started":1}
W20260811 04:37:20.214504 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:20.208860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:52511 (local address 127.2.238.190:40775)
0811 04:37:20.209038 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:20.209041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:20.209123 (+ 82us) server_negotiation.cc:407] Connection header received
0811 04:37:20.209228 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:20.209234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:20.209279 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:20.209346 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:20.209859 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.210497 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.211571 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.211808 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.211889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.214393 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:52511: BlockingRecv error: recv got EOF from 127.2.238.130:52511 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:20 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:20.434043 12952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:20.428536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:60805)
0811 04:37:20.428878 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:20.428892 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:20.428997 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:20.429265 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:20.429268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:20.429456 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:20.429574 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.429580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.430348 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.430351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.430935 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:20.430943 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.431317 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.431319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:20.431378 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.431883 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:20.431894 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:20.433854 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":146,"threads_started":1}
W20260811 04:37:20.434108 12950 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:20.428644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:60805 (local address 127.2.238.190:40775)
0811 04:37:20.428833 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:20.428837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:20.428963 (+ 126us) server_negotiation.cc:407] Connection header received
0811 04:37:20.429096 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:20.429098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:20.429144 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:20.429215 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:20.429701 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.430241 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.431034 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:20.431228 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:20.431295 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:20.433962 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:60805: BlockingRecv error: recv got EOF from 127.2.238.131:60805 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:21.040468 12953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:21.034964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:35941)
0811 04:37:21.035267 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:21.035280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:21.035368 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:21.035680 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:21.035684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:21.035887 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:21.035999 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.036006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.036780 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.036783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.037350 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.037358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.037859 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.037864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.037937 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.038452 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:21.038464 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:21.040351 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
W20260811 04:37:21.040628 12954 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:21.035088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:35941 (local address 127.2.238.190:40775)
0811 04:37:21.035408 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:21.035411 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:21.035424 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:37:21.035462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:21.035464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:21.035505 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:21.035591 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:21.036123 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.036657 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.037468 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.037693 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.037768 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.040489 (+ 2721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:35941: BlockingRecv error: recv got EOF from 127.2.238.129:35941 (error 108)
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":101,"threads_started":1}
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:21.220849 12955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:21.215299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:51165)
0811 04:37:21.215592 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:21.215605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:21.215731 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:21.216083 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:21.216086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:21.216271 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:21.216375 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.216381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.217163 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.217166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.217709 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.217717 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.218251 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.218254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.218321 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.218785 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:21.218796 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:21.220736 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":90,"threads_started":1}
W20260811 04:37:21.221112 12954 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:21.215363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:51165 (local address 127.2.238.190:40775)
0811 04:37:21.215496 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:21.215501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:21.215714 (+ 213us) server_negotiation.cc:407] Connection header received
0811 04:37:21.215799 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:21.215801 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:21.215851 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:21.215929 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:21.216489 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.217035 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.217871 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.218128 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.218217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.220886 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:51165: BlockingRecv error: recv got EOF from 127.2.238.130:51165 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:21 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.131@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:21.443843 12956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:21.434850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.131:55871)
0811 04:37:21.435127 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:21.435140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:21.435237 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:21.435468 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:21.435470 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:21.435651 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:21.435748 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.435753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.436460 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.436463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.437025 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:21.437034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.437452 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.437454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:21.437515 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.441907 (+ 4392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:21.441920 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:21.443723 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":107,"threads_started":1}
W20260811 04:37:21.444000 12954 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:21.434905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:55871 (local address 127.2.238.190:40775)
0811 04:37:21.435038 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:21.435049 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:21.435189 (+ 140us) server_negotiation.cc:407] Connection header received
0811 04:37:21.435317 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:21.435319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:21.435363 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:21.435426 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:21.435841 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.436371 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.437138 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:21.437354 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:21.437432 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:21.443853 (+ 6421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.131:55871: BlockingRecv error: recv got EOF from 127.2.238.131:55871 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 11 04:37:22 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:22 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.129@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:22.047183 12957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:22.041464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.129:39847)
0811 04:37:22.041750 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:22.041764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:22.041914 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:22.042185 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:22.042187 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:22.042370 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:22.042490 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.042496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.043241 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.043244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.043913 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.043925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.044417 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.044421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.044506 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.045071 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:22.045083 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:22.047068 (+ 1985us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":99,"threads_started":1}
W20260811 04:37:22.047339 12958 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:22.041591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:39847 (local address 127.2.238.190:40775)
0811 04:37:22.041842 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:22.041846 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:22.041867 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:37:22.042002 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:22.042004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:22.042049 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:22.042154 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:22.042595 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.043139 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.044044 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.044299 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.044412 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.047208 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.129:39847: BlockingRecv error: recv got EOF from 127.2.238.129:39847 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":87,"threads_started":1}
Aug 11 04:37:22 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
Aug 11 04:37:22 dist-test-slave-0kls krb5kdc[11843](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.238.130@KRBTEST.COM for oryx/127.2.238.190@KRBTEST.COM, Server not found in Kerberos database
I20260811 04:37:22.227363 12959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:22.221745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:40775 (local address 127.2.238.130:33973)
0811 04:37:22.222103 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:22.222118 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:22.222259 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:22.222560 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:22.222563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:22.222816 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:22.222914 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.222920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.223700 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.223704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.224259 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.224267 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.224736 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.224740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.224846 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.225427 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:22.225439 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:22.227195 (+ 1756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.238.190:40775: Server oryx/127.2.238.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":87,"threads_started":1}
W20260811 04:37:22.227460 12958 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:22.221891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:33973 (local address 127.2.238.190:40775)
0811 04:37:22.222064 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:22.222069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:22.222208 (+ 139us) server_negotiation.cc:407] Connection header received
0811 04:37:22.222375 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:22.222379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:22.222435 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:22.222531 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:22.223046 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.223576 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.224388 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.224611 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.224719 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.227343 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.238.130:33973: BlockingRecv error: recv got EOF from 127.2.238.130:33973 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:37:22.331082 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12453
I20260811 04:37:22.342413 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12615
I20260811 04:37:22.349413 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12755
I20260811 04:37:22.355937 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12380
2026-08-11T04:37:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17976 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-11T04:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:22Z Disabled control of system clock
I20260811 04:37:22.389513 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:45107
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:45107
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260811 04:37:22.502547 12963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:22.502812 12963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:22.502849 12963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:22.506981 12963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:22.507045 12963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:22.507066 12963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:22.507150 12963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:22.507189 12963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:22.511911 12963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:45107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:45107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:22.513005 12963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:22.514544 12963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:22.524029 12971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.524148 12963 server_base.cc:1034] running on GCE node
W20260811 04:37:22.524084 12969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:22.524049 12968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.524958 12963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:22.526209 12963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:22.527407 12963 hybrid_clock.cc:648] HybridClock initialized: now 1786423042527378 us; error 42 us; skew 500 ppm
I20260811 04:37:22.529294 12963 webserver.cc:460] Webserver started at http://127.2.238.190:38441/ using document root <none> and password file <none>
I20260811 04:37:22.530009 12963 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:22.530107 12963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:22.530361 12963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:22.532636 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "816eecac6ebe4b39b3c0d48509424eb8"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "3a966ed2c3e943176474a483d633b167"
server_key_iv: "e89ccf246f578d86283a08201be492c5"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.533277 12963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "816eecac6ebe4b39b3c0d48509424eb8"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "3a966ed2c3e943176474a483d633b167"
server_key_iv: "e89ccf246f578d86283a08201be492c5"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.537330 12963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:37:22.540066 12977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:22.541319 12963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:37:22.541471 12963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "816eecac6ebe4b39b3c0d48509424eb8"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "3a966ed2c3e943176474a483d633b167"
server_key_iv: "e89ccf246f578d86283a08201be492c5"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.541608 12963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:22.563215 12963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:22.563941 12963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:22.564175 12963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:22.572468 12963 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:45107
I20260811 04:37:22.572494 13030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:45107 every 8 connection(s)
I20260811 04:37:22.573513 12963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:22.575482 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 12963
I20260811 04:37:22.575589 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:37:22.575930 3002 external_mini_cluster.cc:1428] Setting key 10bc44f8e9c3693d4e5e8ea9fc199b4d
I20260811 04:37:22.576773 13031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:22.583927 13031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8: Bootstrap starting.
I20260811 04:37:22.586438 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:22.577425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.2.238.190:45107)
0811 04:37:22.578119 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:22.578138 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:22.578185 (+ 47us) server_negotiation.cc:407] Connection header received
0811 04:37:22.578974 (+ 789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:22.578997 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:22.579509 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:22.579926 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:22.580373 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.581876 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.582752 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.583473 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.583634 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.584516 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:22.584521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:22.584533 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:22.584923 (+ 390us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:22.584976 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:22.584996 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:22.585095 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:22.585304 (+ 209us) server_negotiation.cc:299] Negotiation successful
0811 04:37:22.585625 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260811 04:37:22.586539 13031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:22.587433 13031 log.cc:824] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:22.589484 13031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8: No bootstrap required, opened a new log
I20260811 04:37:22.593097 13031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } }
I20260811 04:37:22.593314 13031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:22.593389 13031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 816eecac6ebe4b39b3c0d48509424eb8, State: Initialized, Role: FOLLOWER
I20260811 04:37:22.593875 13031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } }
I20260811 04:37:22.594024 13031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:22.594089 13031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:22.594187 13031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:22.595029 13031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } }
I20260811 04:37:22.595372 13031 leader_election.cc:302] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 816eecac6ebe4b39b3c0d48509424eb8; no voters:
I20260811 04:37:22.595705 13031 leader_election.cc:288] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:37:22.595803 13036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:22.596134 13036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [term 1 LEADER]: Becoming Leader. State: Replica: 816eecac6ebe4b39b3c0d48509424eb8, State: Running, Role: LEADER
I20260811 04:37:22.596521 13036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } }
I20260811 04:37:22.597025 13031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:22.598518 13038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 816eecac6ebe4b39b3c0d48509424eb8. Latest consensus state: current_term: 1 leader_uuid: "816eecac6ebe4b39b3c0d48509424eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } } }
I20260811 04:37:22.598675 13038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:22.598510 13037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "816eecac6ebe4b39b3c0d48509424eb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "816eecac6ebe4b39b3c0d48509424eb8" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 45107 } } }
I20260811 04:37:22.598742 13037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8 [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:22.599066 13047 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:22.601987 13047 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:22.607229 13047 catalog_manager.cc:1347] Generated new cluster ID: 0ec09bb190e9447f9efe3dcc3bf9197a
I20260811 04:37:22.607293 13047 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:22.617868 13047 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:37:22.619073 13047 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:22.635692 13047 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 816eecac6ebe4b39b3c0d48509424eb8: Generated new TSK 0
I20260811 04:37:22.636490 13047 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:37:22.648319 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--builtin_ntp_servers=127.2.238.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260811 04:37:22.764567 13056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:22.764868 13056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:22.764940 13056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:22.768973 13056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:22.769068 13056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:22.769135 13056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:22.773808 13056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:22.774753 13056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:22.776065 13056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:22.785701 13064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:22.785701 13061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.785707 13056 server_base.cc:1034] running on GCE node
W20260811 04:37:22.785707 13062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.786391 13056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:22.786936 13056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:22.788105 13056 hybrid_clock.cc:648] HybridClock initialized: now 1786423042788081 us; error 25 us; skew 500 ppm
I20260811 04:37:22.789992 13056 webserver.cc:460] Webserver started at http://127.2.238.129:45843/ using document root <none> and password file <none>
I20260811 04:37:22.790555 13056 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:22.790643 13056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:22.790875 13056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:22.792995 13056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "3995123e4cef4126a6851c3f47d68c37"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "f16bd4fd29fa4fc457af8ffaa48d9f9d"
server_key_iv: "f82ea0754225d4c21102de3d74743d2e"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.793555 13056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "3995123e4cef4126a6851c3f47d68c37"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "f16bd4fd29fa4fc457af8ffaa48d9f9d"
server_key_iv: "f82ea0754225d4c21102de3d74743d2e"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.797283 13056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:37:22.799927 13070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:22.801123 13056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:22.801268 13056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "3995123e4cef4126a6851c3f47d68c37"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "f16bd4fd29fa4fc457af8ffaa48d9f9d"
server_key_iv: "f82ea0754225d4c21102de3d74743d2e"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.801375 13056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:22.823359 13056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:22.823997 13056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:22.824182 13056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:22.824735 13056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:22.825728 13056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:22.825811 13056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:22.825906 13056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:22.825956 13056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:22.834615 13056 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44797
I20260811 04:37:22.834638 13191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44797 every 8 connection(s)
I20260811 04:37:22.835505 13056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:22.844640 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:22.837252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56179 (local address 127.2.238.190:45107)
0811 04:37:22.837383 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:22.837387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:22.838488 (+ 1101us) server_negotiation.cc:407] Connection header received
0811 04:37:22.839873 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:22.839876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:22.839938 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:22.840049 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:22.840942 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.841478 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.842900 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.843105 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.843197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.844197 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:22.844200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:22.844202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:22.844214 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:22.844231 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:22.844239 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:22.844306 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:22.844469 (+ 163us) server_negotiation.cc:299] Negotiation successful
0811 04:37:22.844527 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:37:22.845456 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:22.837635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:45107 (local address 127.2.238.129:56179)
0811 04:37:22.838342 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:22.838383 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:22.839644 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:22.840237 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:22.840245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:22.840325 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:22.840795 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.840809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.841621 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.841625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.842752 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:22.842764 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:22.843263 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:22.843266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:22.843394 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:22.843912 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:22.843913 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:22.843955 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:22.844033 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:22.844317 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:22.844324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:22.844328 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:37:22.844422 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 04:37:22.844632 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":189,"threads_started":1}
I20260811 04:37:22.845763 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13056
I20260811 04:37:22.845896 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:37:22.846280 3002 external_mini_cluster.cc:1428] Setting key db41fed703d065ee7d85a5d08ea7b5b7
I20260811 04:37:22.846748 13192 heartbeater.cc:347] Connected to a master server at 127.2.238.190:45107
I20260811 04:37:22.846988 13192 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:22.847548 13192 heartbeater.cc:511] Master 127.2.238.190:45107 requested a full tablet report, sending...
I20260811 04:37:22.848307 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--builtin_ntp_servers=127.2.238.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 04:37:22.849246 12994 ts_manager.cc:194] Registered new tserver with Master: 3995123e4cef4126a6851c3f47d68c37 (127.2.238.129:44797)
I20260811 04:37:22.850994 12994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.129:56179
W20260811 04:37:22.960316 13195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:22.960543 13195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:22.960579 13195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:22.964331 13195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:22.964375 13195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:22.964418 13195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:22.968955 13195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:22.969920 13195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:22.971292 13195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:22.980647 13202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:22.980652 13201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.980765 13195 server_base.cc:1034] running on GCE node
W20260811 04:37:22.980652 13204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:22.981350 13195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:22.982009 13195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:22.983201 13195 hybrid_clock.cc:648] HybridClock initialized: now 1786423042983187 us; error 25 us; skew 500 ppm
I20260811 04:37:22.984854 13195 webserver.cc:460] Webserver started at http://127.2.238.130:34013/ using document root <none> and password file <none>
I20260811 04:37:22.985419 13195 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:22.985510 13195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:22.985743 13195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:22.987684 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "61383e0be7034606a997f9c40748708d"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "67fbc2996fc5d48a72ab5cdbd1b476f3"
server_key_iv: "c22bf1a77607a74120f64d44e0761497"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.988246 13195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "61383e0be7034606a997f9c40748708d"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "67fbc2996fc5d48a72ab5cdbd1b476f3"
server_key_iv: "c22bf1a77607a74120f64d44e0761497"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.991653 13195 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 04:37:22.994184 13210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:22.995186 13195 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:22.995307 13195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "61383e0be7034606a997f9c40748708d"
format_stamp: "Formatted at 2026-08-11 04:37:22 on dist-test-slave-0kls"
server_key: "67fbc2996fc5d48a72ab5cdbd1b476f3"
server_key_iv: "c22bf1a77607a74120f64d44e0761497"
server_key_version: "encryptionkey@0"
I20260811 04:37:22.995404 13195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:23.005745 13195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:23.006372 13195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:23.006539 13195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:23.007071 13195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:23.007906 13195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:23.007972 13195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.008033 13195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:23.008082 13195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.016984 13195 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:33479
I20260811 04:37:23.017005 13335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:33479 every 8 connection(s)
I20260811 04:37:23.017822 13195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:23.025892 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13195
I20260811 04:37:23.025993 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:37:23.026254 3002 external_mini_cluster.cc:1428] Setting key 4dd1e8b345effea0588176f1fb9e5cd9
I20260811 04:37:23.026974 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.019637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:43035 (local address 127.2.238.190:45107)
0811 04:37:23.019798 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.019802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.020999 (+ 1197us) server_negotiation.cc:407] Connection header received
0811 04:37:23.022468 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.022470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.022520 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.022603 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.023498 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.024001 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.025077 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.025273 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.025349 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:23.026448 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:23.026452 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:23.026454 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:23.026467 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:23.026489 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:23.026498 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:23.026575 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:23.026758 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 04:37:23.026811 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:37:23.027822 13338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.020050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:45107 (local address 127.2.238.130:43035)
0811 04:37:23.020857 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:23.020897 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:23.022272 (+ 1375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:23.022760 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:23.022767 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:23.022864 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:23.023366 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:23.023379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.024117 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.024119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:23.024962 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:23.024971 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.025401 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.025404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:23.025533 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:23.026189 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:23.026190 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:23.026226 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:23.026282 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:23.026590 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:23.026598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:23.026602 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:37:23.026724 (+ 122us) client_negotiation.cc:240] Negotiation successful
0811 04:37:23.027009 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":118,"threads_started":1}
I20260811 04:37:23.028491 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--builtin_ntp_servers=127.2.238.148:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260811 04:37:23.028959 13336 heartbeater.cc:347] Connected to a master server at 127.2.238.190:45107
I20260811 04:37:23.029186 13336 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:23.029799 13336 heartbeater.cc:511] Master 127.2.238.190:45107 requested a full tablet report, sending...
I20260811 04:37:23.031002 12994 ts_manager.cc:194] Registered new tserver with Master: 61383e0be7034606a997f9c40748708d (127.2.238.130:33479)
I20260811 04:37:23.031848 12994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.130:43035
W20260811 04:37:23.139530 13339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:23.139801 13339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:23.139914 13339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:23.143718 13339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:23.143782 13339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:23.143852 13339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:23.148461 13339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:45107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:23.149380 13339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:23.150692 13339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:23.159690 13345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.159690 13348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.159842 13344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.160149 13339 server_base.cc:1034] running on GCE node
I20260811 04:37:23.160550 13339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:23.161175 13339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:23.162375 13339 hybrid_clock.cc:648] HybridClock initialized: now 1786423043162361 us; error 26 us; skew 500 ppm
I20260811 04:37:23.164143 13339 webserver.cc:460] Webserver started at http://127.2.238.131:34611/ using document root <none> and password file <none>
I20260811 04:37:23.164695 13339 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:23.164783 13339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:23.165030 13339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:23.167351 13339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "d946708738ec45ca98d7fa580dad6427"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "7e8e1a4f2975832866897be34213a632"
server_key_iv: "9fa61002775e6095fbf1b9032d23e5d3"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.167858 13339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "d946708738ec45ca98d7fa580dad6427"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "7e8e1a4f2975832866897be34213a632"
server_key_iv: "9fa61002775e6095fbf1b9032d23e5d3"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.171479 13339 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:37:23.174556 13354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.175805 13339 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260811 04:37:23.175940 13339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "d946708738ec45ca98d7fa580dad6427"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "7e8e1a4f2975832866897be34213a632"
server_key_iv: "9fa61002775e6095fbf1b9032d23e5d3"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.176059 13339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:23.188681 13339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:23.189257 13339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:23.189428 13339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:23.189990 13339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:23.190860 13339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:23.190928 13339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.190994 13339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:23.191046 13339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.200237 13339 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:33107
I20260811 04:37:23.200269 13478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:33107 every 8 connection(s)
I20260811 04:37:23.201089 13339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:23.206298 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13339
I20260811 04:37:23.206391 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:37:23.206652 3002 external_mini_cluster.cc:1428] Setting key 54a43065035fa9024ca351c968398c18
I20260811 04:37:23.210314 13034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.202825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:49001 (local address 127.2.238.190:45107)
0811 04:37:23.202979 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.202982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.203862 (+ 880us) server_negotiation.cc:407] Connection header received
0811 04:37:23.205191 (+ 1329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.205194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.205238 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.205306 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.206214 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.206900 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.208477 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.208783 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.208884 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:23.209905 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:23.209907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:23.209909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:23.209919 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:23.209934 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:23.209940 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:23.209990 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:23.210155 (+ 165us) server_negotiation.cc:299] Negotiation successful
0811 04:37:23.210201 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260811 04:37:23.211050 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.203193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:45107 (local address 127.2.238.131:49001)
0811 04:37:23.203753 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:23.203789 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:23.205011 (+ 1222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:23.205421 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:23.205427 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:23.205500 (+ 73us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:23.206078 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:23.206091 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.207033 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.207036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:23.208365 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:23.208376 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:23.208927 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:23.208930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:23.209063 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:23.209628 (+ 565us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:23.209630 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:23.209665 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:23.209720 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:23.210006 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:23.210013 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:23.210017 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:37:23.210103 (+ 86us) client_negotiation.cc:240] Negotiation successful
0811 04:37:23.210317 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260811 04:37:23.212150 13479 heartbeater.cc:347] Connected to a master server at 127.2.238.190:45107
I20260811 04:37:23.212380 13479 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:23.212918 13479 heartbeater.cc:511] Master 127.2.238.190:45107 requested a full tablet report, sending...
I20260811 04:37:23.213897 12994 ts_manager.cc:194] Registered new tserver with Master: d946708738ec45ca98d7fa580dad6427 (127.2.238.131:33107)
I20260811 04:37:23.214414 12994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.131:49001
I20260811 04:37:23.220294 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 04:37:23.222988 13034 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:23.222231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.2.238.190:45107)
0811 04:37:23.222350 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.222354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.222515 (+ 161us) server_negotiation.cc:407] Connection header received
0811 04:37:23.222615 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.222617 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.222652 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.222718 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.222902 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45136: BlockingRecv error: recv got EOF from 127.0.0.1:45136 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:37:23.223163 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13056
I20260811 04:37:23.231384 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13195
I20260811 04:37:23.241998 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13339
I20260811 04:37:23.249984 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 12963
2026-08-11T04:37:23Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (892 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-11T04:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:23Z Disabled control of system clock
I20260811 04:37:23.281294 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43235
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:43235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 04:37:23.391710 13491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:23.391944 13491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:23.392004 13491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:23.395617 13491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:23.395661 13491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:23.395680 13491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:23.395695 13491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:23.395709 13491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:23.400058 13491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:43235
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:23.401028 13491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:23.402395 13491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:23.410481 13496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.410494 13499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.410573 13497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.410807 13491 server_base.cc:1034] running on GCE node
I20260811 04:37:23.411350 13491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:23.412390 13491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:23.413581 13491 hybrid_clock.cc:648] HybridClock initialized: now 1786423043413565 us; error 35 us; skew 500 ppm
I20260811 04:37:23.415330 13491 webserver.cc:460] Webserver started at http://127.2.238.190:33871/ using document root <none> and password file <none>
I20260811 04:37:23.415870 13491 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:23.415983 13491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:23.416215 13491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:23.418203 13491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "3cd89f09178c42fd968bf56f020f983a"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "adeeddc6ea7c1d61aeb720202ebffbfe"
server_key_iv: "7ab806606fd0a6215eddf32c93dd09c7"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.418756 13491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "3cd89f09178c42fd968bf56f020f983a"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "adeeddc6ea7c1d61aeb720202ebffbfe"
server_key_iv: "7ab806606fd0a6215eddf32c93dd09c7"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.422575 13491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:37:23.424902 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.426093 13491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:23.426250 13491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "3cd89f09178c42fd968bf56f020f983a"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "adeeddc6ea7c1d61aeb720202ebffbfe"
server_key_iv: "7ab806606fd0a6215eddf32c93dd09c7"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.426409 13491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:23.430655 13491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:23.431248 13491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:23.431428 13491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:23.438814 13491 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:43235
I20260811 04:37:23.438828 13561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:43235 every 8 connection(s)
I20260811 04:37:23.439713 13491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:23.442866 13562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:23.446786 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13491
I20260811 04:37:23.446921 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:37:23.447193 3002 external_mini_cluster.cc:1428] Setting key 87c4f7ecc056374b849d0a0a0495d1d4
I20260811 04:37:23.449297 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a: Bootstrap starting.
I20260811 04:37:23.451890 13562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:23.452615 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.448455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44010 (local address 127.2.238.190:43235)
0811 04:37:23.449018 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.449031 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.449063 (+ 32us) server_negotiation.cc:407] Connection header received
0811 04:37:23.449822 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.449853 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.450201 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.450522 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.451313 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:23.451317 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:23.451328 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:23.451338 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:23.451392 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:23.451407 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:23.451541 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:23.451637 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 04:37:23.451895 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":107,"threads_started":1}
I20260811 04:37:23.452644 13562 log.cc:824] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:23.454901 13562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a: No bootstrap required, opened a new log
I20260811 04:37:23.458616 13562 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } }
I20260811 04:37:23.458910 13562 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:23.458976 13562 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd89f09178c42fd968bf56f020f983a, State: Initialized, Role: FOLLOWER
I20260811 04:37:23.459376 13562 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } }
I20260811 04:37:23.459497 13562 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:23.459542 13562 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:23.459612 13562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:23.460479 13562 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } }
I20260811 04:37:23.460764 13562 leader_election.cc:302] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cd89f09178c42fd968bf56f020f983a; no voters:
I20260811 04:37:23.461040 13562 leader_election.cc:288] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:37:23.461153 13568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:23.461441 13568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [term 1 LEADER]: Becoming Leader. State: Replica: 3cd89f09178c42fd968bf56f020f983a, State: Running, Role: LEADER
I20260811 04:37:23.461769 13568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } }
I20260811 04:37:23.462404 13562 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:23.463765 13571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cd89f09178c42fd968bf56f020f983a. Latest consensus state: current_term: 1 leader_uuid: "3cd89f09178c42fd968bf56f020f983a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } } }
I20260811 04:37:23.463907 13571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:23.463762 13569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cd89f09178c42fd968bf56f020f983a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cd89f09178c42fd968bf56f020f983a" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43235 } } }
I20260811 04:37:23.463958 13569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:23.464615 13578 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:23.467458 13578 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:23.472949 13578 catalog_manager.cc:1347] Generated new cluster ID: a71889731f1e4fe2a7d90009df6b50d0
I20260811 04:37:23.473003 13578 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:23.482532 13578 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:37:23.482832 13578 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:23.490975 13578 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cd89f09178c42fd968bf56f020f983a: Generated new TSK 0
I20260811 04:37:23.491705 13578 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:37:23.506098 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--builtin_ntp_servers=127.2.238.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260811 04:37:23.621146 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:23.621447 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:23.621538 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:23.625708 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:23.625849 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:23.625973 13588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:23.630759 13588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:23.632001 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:23.633554 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:23.643388 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.643374 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.643471 13588 server_base.cc:1034] running on GCE node
W20260811 04:37:23.643374 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.644058 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:23.644677 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:23.645864 13588 hybrid_clock.cc:648] HybridClock initialized: now 1786423043645866 us; error 55 us; skew 500 ppm
I20260811 04:37:23.647565 13588 webserver.cc:460] Webserver started at http://127.2.238.129:42837/ using document root <none> and password file <none>
I20260811 04:37:23.648118 13588 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:23.648176 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:23.648413 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:23.650458 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "8257426eef2e40b8b8135b8134dec609"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "491a70a9494663f7b10020517879a8d1"
server_key_iv: "98fcb8c0162c8051afc0707519f5eeba"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.651058 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "8257426eef2e40b8b8135b8134dec609"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "491a70a9494663f7b10020517879a8d1"
server_key_iv: "98fcb8c0162c8051afc0707519f5eeba"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.654806 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:37:23.657224 13603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.658447 13588 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:23.658572 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "8257426eef2e40b8b8135b8134dec609"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "491a70a9494663f7b10020517879a8d1"
server_key_iv: "98fcb8c0162c8051afc0707519f5eeba"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.658680 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:23.663269 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:23.663887 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:23.664093 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:23.664649 13588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:23.665549 13588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:23.665617 13588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.665674 13588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:23.665720 13588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.674674 13588 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:37287
I20260811 04:37:23.674711 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:37287 every 8 connection(s)
I20260811 04:37:23.675536 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:23.681682 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.677276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:48841 (local address 127.2.238.190:43235)
0811 04:37:23.677410 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.677414 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.678507 (+ 1093us) server_negotiation.cc:407] Connection header received
0811 04:37:23.679865 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.679868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.679914 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.679994 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.681174 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:23.681177 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:23.681180 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:23.681195 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:23.681222 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:23.681230 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:23.681331 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:23.681491 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 04:37:23.681539 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:37:23.682546 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.677620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:43235 (local address 127.2.238.129:48841)
0811 04:37:23.678355 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:23.678395 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:23.679629 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:23.680161 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:23.680168 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:23.680253 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:23.680811 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:23.680812 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:23.680851 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:23.680918 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:23.681326 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:23.681336 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:23.681344 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:37:23.681528 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 04:37:23.681821 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":167,"threads_started":1}
I20260811 04:37:23.683331 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13588
I20260811 04:37:23.683444 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:37:23.683745 3002 external_mini_cluster.cc:1428] Setting key 63305a83636c49dd9b2a0a7b525382fb
I20260811 04:37:23.683871 13724 heartbeater.cc:347] Connected to a master server at 127.2.238.190:43235
I20260811 04:37:23.684105 13724 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:23.684527 13724 heartbeater.cc:511] Master 127.2.238.190:43235 requested a full tablet report, sending...
I20260811 04:37:23.685974 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--builtin_ntp_servers=127.2.238.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 04:37:23.686188 13524 ts_manager.cc:194] Registered new tserver with Master: 8257426eef2e40b8b8135b8134dec609 (127.2.238.129:37287)
W20260811 04:37:23.801299 13727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:23.801594 13727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:23.801635 13727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:23.805747 13727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:23.805835 13727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:23.805895 13727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:23.810917 13727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:23.812160 13727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:23.813652 13727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:23.824020 13733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.824020 13735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.824081 13732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.824311 13727 server_base.cc:1034] running on GCE node
I20260811 04:37:23.824817 13727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:23.825502 13727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:23.826673 13727 hybrid_clock.cc:648] HybridClock initialized: now 1786423043826634 us; error 48 us; skew 500 ppm
I20260811 04:37:23.828507 13727 webserver.cc:460] Webserver started at http://127.2.238.130:39447/ using document root <none> and password file <none>
I20260811 04:37:23.829092 13727 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:23.829150 13727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:23.829399 13727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:23.831585 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "ca22282ef00d4e44999a846b8ede1143"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "651e84d23a78c164c9e80894339059e8"
server_key_iv: "ceda546a6f06a166ee88e0ef662b362d"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.832149 13727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "ca22282ef00d4e44999a846b8ede1143"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "651e84d23a78c164c9e80894339059e8"
server_key_iv: "ceda546a6f06a166ee88e0ef662b362d"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.835919 13727 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 04:37:23.838519 13742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.839730 13727 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:37:23.839905 13727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "ca22282ef00d4e44999a846b8ede1143"
format_stamp: "Formatted at 2026-08-11 04:37:23 on dist-test-slave-0kls"
server_key: "651e84d23a78c164c9e80894339059e8"
server_key_iv: "ceda546a6f06a166ee88e0ef662b362d"
server_key_version: "encryptionkey@0"
I20260811 04:37:23.840021 13727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:23.844635 13727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:23.845489 13727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:23.845701 13727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:23.846306 13727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:23.847194 13727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:23.847266 13727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.847327 13727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:23.847378 13727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:23.855894 13727 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:34261
I20260811 04:37:23.855926 13861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:34261 every 8 connection(s)
I20260811 04:37:23.856730 13727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:23.862900 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.858570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:50255 (local address 127.2.238.190:43235)
0811 04:37:23.858731 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:23.858736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:23.859801 (+ 1065us) server_negotiation.cc:407] Connection header received
0811 04:37:23.861029 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:23.861032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:23.861075 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:23.861149 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:23.862350 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:23.862354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:23.862356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:23.862369 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:23.862393 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:23.862407 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:23.862506 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:23.862701 (+ 195us) server_negotiation.cc:299] Negotiation successful
0811 04:37:23.862751 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:37:23.863626 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13727
I20260811 04:37:23.863770 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:37:23.864012 13865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:23.859002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:43235 (local address 127.2.238.130:50255)
0811 04:37:23.859642 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:23.859681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:23.860844 (+ 1163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:23.861316 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:23.861322 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:23.861430 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:23.862037 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:23.862038 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:23.862098 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:23.862167 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:23.862503 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:23.862514 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:23.862524 (+ 10us) client_negotiation.cc:769] Sending connection context
0811 04:37:23.862732 (+ 208us) client_negotiation.cc:240] Negotiation successful
0811 04:37:23.862976 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":115,"threads_started":1}
I20260811 04:37:23.864073 3002 external_mini_cluster.cc:1428] Setting key 4f34aef81052eb4ee3c222be19ba73c2
I20260811 04:37:23.865198 13862 heartbeater.cc:347] Connected to a master server at 127.2.238.190:43235
I20260811 04:37:23.865458 13862 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:23.865938 13862 heartbeater.cc:511] Master 127.2.238.190:43235 requested a full tablet report, sending...
I20260811 04:37:23.866094 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--builtin_ntp_servers=127.2.238.148:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260811 04:37:23.867080 13524 ts_manager.cc:194] Registered new tserver with Master: ca22282ef00d4e44999a846b8ede1143 (127.2.238.130:34261)
W20260811 04:37:23.978961 13866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:23.979197 13866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:23.979233 13866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:23.983245 13866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:23.983300 13866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:23.983353 13866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:23.987875 13866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:43235
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:23.988795 13866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:23.990094 13866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:23.999533 13874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.999541 13871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:23.999552 13872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:23.999979 13866 server_base.cc:1034] running on GCE node
I20260811 04:37:24.000385 13866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:24.001017 13866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:24.002228 13866 hybrid_clock.cc:648] HybridClock initialized: now 1786423044002191 us; error 28 us; skew 500 ppm
I20260811 04:37:24.004310 13866 webserver.cc:460] Webserver started at http://127.2.238.131:35357/ using document root <none> and password file <none>
I20260811 04:37:24.005163 13866 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:24.005270 13866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:24.005502 13866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:24.008442 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "16613942b4c84e9ab6e91f5d4cf9b939"
format_stamp: "Formatted at 2026-08-11 04:37:24 on dist-test-slave-0kls"
server_key: "c2ccc14624a85354c743bc42734b6697"
server_key_iv: "aa2982f7ddcebb758f087d44be9057a8"
server_key_version: "encryptionkey@0"
I20260811 04:37:24.009212 13866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "16613942b4c84e9ab6e91f5d4cf9b939"
format_stamp: "Formatted at 2026-08-11 04:37:24 on dist-test-slave-0kls"
server_key: "c2ccc14624a85354c743bc42734b6697"
server_key_iv: "aa2982f7ddcebb758f087d44be9057a8"
server_key_version: "encryptionkey@0"
I20260811 04:37:24.013639 13866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260811 04:37:24.016006 13880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:24.017169 13866 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:24.017289 13866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "16613942b4c84e9ab6e91f5d4cf9b939"
format_stamp: "Formatted at 2026-08-11 04:37:24 on dist-test-slave-0kls"
server_key: "c2ccc14624a85354c743bc42734b6697"
server_key_iv: "aa2982f7ddcebb758f087d44be9057a8"
server_key_version: "encryptionkey@0"
I20260811 04:37:24.017413 13866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:24.022256 13866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:24.022866 13866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:24.023097 13866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:24.023670 13866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:24.024653 13866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:24.024705 13866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:24.024776 13866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:24.024816 13866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:24.033847 13866 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:42443
I20260811 04:37:24.033877 14004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:42443 every 8 connection(s)
I20260811 04:37:24.034788 13866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:24.040932 13565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:24.036558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:56201 (local address 127.2.238.190:43235)
0811 04:37:24.036686 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:24.036689 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:24.037818 (+ 1129us) server_negotiation.cc:407] Connection header received
0811 04:37:24.039083 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:24.039087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:24.039131 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:24.039234 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:24.040435 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:24.040438 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:24.040443 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:37:24.040461 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:24.040481 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:37:24.040489 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:24.040590 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:24.040737 (+ 147us) server_negotiation.cc:299] Negotiation successful
0811 04:37:24.040796 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:24.041857 14008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:24.036935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:43235 (local address 127.2.238.131:56201)
0811 04:37:24.037614 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:24.037650 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:24.038883 (+ 1233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:24.039398 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:24.039405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:24.039485 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:24.040110 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:37:24.040112 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:24.040172 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:37:24.040259 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:24.040572 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:24.040581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:24.040587 (+ 6us) client_negotiation.cc:769] Sending connection context
0811 04:37:24.040768 (+ 181us) client_negotiation.cc:240] Negotiation successful
0811 04:37:24.041000 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260811 04:37:24.043004 14006 heartbeater.cc:347] Connected to a master server at 127.2.238.190:43235
I20260811 04:37:24.043200 14006 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:24.043556 14006 heartbeater.cc:511] Master 127.2.238.190:43235 requested a full tablet report, sending...
I20260811 04:37:24.044341 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 13866
I20260811 04:37:24.044467 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:37:24.044704 13524 ts_manager.cc:194] Registered new tserver with Master: 16613942b4c84e9ab6e91f5d4cf9b939 (127.2.238.131:42443)
I20260811 04:37:24.044924 3002 external_mini_cluster.cc:1428] Setting key e8e6eb6c0e82797eed69966859614cbd
I20260811 04:37:24.046904 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260811 04:37:24.049610 13565 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:37:24.048776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.2.238.190:43235)
0811 04:37:24.048938 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:24.048944 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:24.049068 (+ 124us) server_negotiation.cc:407] Connection header received
0811 04:37:24.049163 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:24.049165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:24.049196 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:24.049260 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:24.049458 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44018: BlockingRecv error: recv got EOF from 127.0.0.1:44018 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260811 04:37:24.049703 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13588
I20260811 04:37:24.057720 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13727
I20260811 04:37:24.065973 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13866
I20260811 04:37:24.071992 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 13491
2026-08-11T04:37:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (822 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:37:24 dist-test-slave-0kls krb5kdc[14016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:37:24 dist-test-slave-0kls krb5kdc[14016](info): set up 2 sockets
Aug 11 04:37:24 dist-test-slave-0kls krb5kdc[14016](info): commencing operation
krb5kdc: starting...
W20260811 04:37:25.116241 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:37:25.240412 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41117
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:41117
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:25.353660 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:25.353940 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:25.353979 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:25.358071 14030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:37:25.358125 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:25.358144 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:25.358161 14030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:37:25.358213 14030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:37:25.363554 14030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:41117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:25.364606 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:25.365653 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:25.374176 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:25.374217 14040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:25.374217 14035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:25.374325 14030 server_base.cc:1034] running on GCE node
I20260811 04:37:25.375001 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:25.376086 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:25.377245 14030 hybrid_clock.cc:648] HybridClock initialized: now 1786423045377219 us; error 36 us; skew 500 ppm
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:25.379907 14030 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:37:25.380722 14030 webserver.cc:460] Webserver started at http://127.2.238.190:39545/ using document root <none> and password file <none>
I20260811 04:37:25.381233 14030 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:25.381287 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:25.381438 14030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:25.383549 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "41eecedc1c2449e08b546e2b858301bc"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "20f27793d2df87f3da145809aa5917d1"
server_key_iv: "11c771f0e2d1255982ff77a226d8cb32"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.384065 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "41eecedc1c2449e08b546e2b858301bc"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "20f27793d2df87f3da145809aa5917d1"
server_key_iv: "11c771f0e2d1255982ff77a226d8cb32"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.387631 14030 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260811 04:37:25.389703 14048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.390913 14030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:37:25.391011 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "41eecedc1c2449e08b546e2b858301bc"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "20f27793d2df87f3da145809aa5917d1"
server_key_iv: "11c771f0e2d1255982ff77a226d8cb32"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.391088 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:25.403504 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:25.406479 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:25.406629 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:25.413293 14030 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:41117
I20260811 04:37:25.413584 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:41117 every 8 connection(s)
I20260811 04:37:25.414291 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:37:25.416885 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:25.417500 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 14030
I20260811 04:37:25.417621 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:37:25.417977 3002 external_mini_cluster.cc:1428] Setting key 0ad85db9f8f5add9f03e722380733dfb
I20260811 04:37:25.423677 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: Bootstrap starting.
I20260811 04:37:25.426286 14106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:25.427037 14106 log.cc:824] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:25.429073 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: No bootstrap required, opened a new log
I20260811 04:37:25.433537 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:25.419357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.2.238.190:41117)
0811 04:37:25.419760 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:25.419808 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:25.419873 (+ 65us) server_negotiation.cc:407] Connection header received
0811 04:37:25.420693 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:25.420717 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:25.421136 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:25.421497 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:25.422252 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.423691 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.424587 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.425310 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.425464 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:25.428226 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:25.428231 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:25.428262 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:25.428288 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:25.430308 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.430767 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.430774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.430780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.430925 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.431260 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.431265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.431267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.431799 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:25.432008 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:25.432284 (+ 276us) server_negotiation.cc:299] Negotiation successful
0811 04:37:25.432578 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":97,"threads_started":1}
I20260811 04:37:25.433676 14106 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } }
I20260811 04:37:25.433966 14106 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:25.434029 14106 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41eecedc1c2449e08b546e2b858301bc, State: Initialized, Role: FOLLOWER
I20260811 04:37:25.434469 14106 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } }
I20260811 04:37:25.434644 14106 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:37:25.434707 14106 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:37:25.434811 14106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:25.435735 14106 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } }
I20260811 04:37:25.436101 14106 leader_election.cc:302] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41eecedc1c2449e08b546e2b858301bc; no voters:
I20260811 04:37:25.436412 14106 leader_election.cc:288] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:37:25.436560 14113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:25.436822 14113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [term 1 LEADER]: Becoming Leader. State: Replica: 41eecedc1c2449e08b546e2b858301bc, State: Running, Role: LEADER
I20260811 04:37:25.437134 14113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } }
I20260811 04:37:25.437651 14106 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:37:25.438769 14115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41eecedc1c2449e08b546e2b858301bc. Latest consensus state: current_term: 1 leader_uuid: "41eecedc1c2449e08b546e2b858301bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } } }
I20260811 04:37:25.438899 14115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:25.439318 14123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:37:25.439265 14114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41eecedc1c2449e08b546e2b858301bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41eecedc1c2449e08b546e2b858301bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41117 } } }
I20260811 04:37:25.439427 14114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc [sys.catalog]: This master's current role is: LEADER
I20260811 04:37:25.442175 14123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:37:25.447691 14123 catalog_manager.cc:1347] Generated new cluster ID: a45c5d327ed34300a9216a8f1da8037c
I20260811 04:37:25.447745 14123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:37:25.459712 14123 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:37:25.460697 14123 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:37:25.472723 14123 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: Generated new TSK 0
I20260811 04:37:25.473584 14123 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:37:25.510788 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:25.625545 14135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:25.625906 14135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:25.625998 14135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:25.630473 14135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:25.630563 14135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:25.630654 14135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:37:25.636101 14135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:25.637274 14135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:25.638516 14135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:25.648502 14141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:25.648526 14140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:25.648645 14135 server_base.cc:1034] running on GCE node
W20260811 04:37:25.648526 14143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:25.649245 14135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:25.650133 14135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:25.651317 14135 hybrid_clock.cc:648] HybridClock initialized: now 1786423045651296 us; error 41 us; skew 500 ppm
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:25.654146 14135 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:37:25.655028 14135 webserver.cc:460] Webserver started at http://127.2.238.129:40643/ using document root <none> and password file <none>
I20260811 04:37:25.655611 14135 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:25.655673 14135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:25.655941 14135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:25.658264 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "0020bf3894e54834a1666b711feb3ed8"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "25798da814e0d72f0c3b9eb5691a2b20"
server_key_iv: "023f023212c6fe8120a7fa5f939d9aaa"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.658941 14135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "0020bf3894e54834a1666b711feb3ed8"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "25798da814e0d72f0c3b9eb5691a2b20"
server_key_iv: "023f023212c6fe8120a7fa5f939d9aaa"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.662829 14135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:37:25.665355 14151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.666638 14135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:25.666769 14135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "0020bf3894e54834a1666b711feb3ed8"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "25798da814e0d72f0c3b9eb5691a2b20"
server_key_iv: "023f023212c6fe8120a7fa5f939d9aaa"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.666908 14135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:25.695467 14135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:25.698521 14135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:25.698726 14135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:25.699365 14135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:25.700240 14135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:25.700289 14135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.700371 14135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:25.700402 14135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.709496 14135 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:33793
I20260811 04:37:25.709522 14274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:33793 every 8 connection(s)
I20260811 04:37:25.710469 14135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:37:25.717926 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 14135
I20260811 04:37:25.718047 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:37:25.718464 3002 external_mini_cluster.cc:1428] Setting key 0f53a7823ecafd052611b49f4330010a
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:25.724985 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:25.712240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:47575 (local address 127.2.238.190:41117)
0811 04:37:25.712397 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:25.712400 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:25.713196 (+ 796us) server_negotiation.cc:407] Connection header received
0811 04:37:25.714721 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:25.714725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:25.714796 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:25.714903 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:25.716077 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.716600 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.717657 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.717977 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.718092 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:25.720879 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:25.720886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:25.720892 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:25.720926 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:25.722536 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.723196 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.723199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.723200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.723278 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.723739 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.723742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.723743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.724031 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:25.724149 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:25.724683 (+ 534us) server_negotiation.cc:299] Negotiation successful
0811 04:37:25.724813 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 04:37:25.725868 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:25.712568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41117 (local address 127.2.238.129:47575)
0811 04:37:25.713028 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:25.713073 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:25.714500 (+ 1427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:25.715088 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:25.715095 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:25.715465 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:25.715927 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:25.715941 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.716726 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.716729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:25.717531 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:25.717540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.718137 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.718141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:25.718373 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:25.719038 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:25.719092 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:25.720705 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:25.722665 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:25.722674 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:25.722691 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:25.723093 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:25.723371 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:25.723375 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:25.723377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:25.723573 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:25.724175 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:25.724201 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:25.724436 (+ 235us) client_negotiation.cc:769] Sending connection context
0811 04:37:25.724639 (+ 203us) client_negotiation.cc:240] Negotiation successful
0811 04:37:25.724959 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":108,"threads_started":1}
I20260811 04:37:25.727241 14275 heartbeater.cc:347] Connected to a master server at 127.2.238.190:41117
I20260811 04:37:25.727494 14275 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:25.727962 14275 heartbeater.cc:511] Master 127.2.238.190:41117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:37:25.729763 14066 ts_manager.cc:194] Registered new tserver with Master: 0020bf3894e54834a1666b711feb3ed8 (127.2.238.129:33793)
I20260811 04:37:25.731019 14066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:47575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:37:25.748900 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:25.860201 14280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:25.860458 14280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:25.860499 14280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:25.865144 14280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:25.865207 14280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:25.865278 14280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:37:25.870487 14280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:25.871619 14280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:25.872623 14280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:25.882510 14288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:25.882521 14285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:25.882510 14286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:25.883051 14280 server_base.cc:1034] running on GCE node
I20260811 04:37:25.883507 14280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:25.884146 14280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:25.885310 14280 hybrid_clock.cc:648] HybridClock initialized: now 1786423045885289 us; error 31 us; skew 500 ppm
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:25.888247 14280 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:37:25.889111 14280 webserver.cc:460] Webserver started at http://127.2.238.130:41285/ using document root <none> and password file <none>
I20260811 04:37:25.889701 14280 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:25.889834 14280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:25.890070 14280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:25.892155 14280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "2d562c7730f54c59a058ec778ed26415"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "e7012b7469cb65e1399fd7fa68a50bf8"
server_key_iv: "31fc75847329d226b2ab87d30b2cbcd7"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.892792 14280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "2d562c7730f54c59a058ec778ed26415"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "e7012b7469cb65e1399fd7fa68a50bf8"
server_key_iv: "31fc75847329d226b2ab87d30b2cbcd7"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.896672 14280 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:37:25.899144 14295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.900333 14280 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:37:25.900476 14280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "2d562c7730f54c59a058ec778ed26415"
format_stamp: "Formatted at 2026-08-11 04:37:25 on dist-test-slave-0kls"
server_key: "e7012b7469cb65e1399fd7fa68a50bf8"
server_key_iv: "31fc75847329d226b2ab87d30b2cbcd7"
server_key_version: "encryptionkey@0"
I20260811 04:37:25.900586 14280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:25.915268 14280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:25.918481 14280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:25.918680 14280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:25.919260 14280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:25.920112 14280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:25.920185 14280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.920258 14280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:25.920312 14280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:25.928725 14280 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:39097
I20260811 04:37:25.928788 14418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:39097 every 8 connection(s)
I20260811 04:37:25.929592 14280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:37:25.935977 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 14280
I20260811 04:37:25.936067 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:37:25.936362 3002 external_mini_cluster.cc:1428] Setting key cd2b015e43e14fcb13b5fdd0428f21d2
Aug 11 04:37:25 dist-test-slave-0kls krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423045, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:25.944193 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:25.931349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:45177 (local address 127.2.238.190:41117)
0811 04:37:25.931477 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:25.931481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:25.932294 (+ 813us) server_negotiation.cc:407] Connection header received
0811 04:37:25.933819 (+ 1525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:25.933823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:25.933880 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:25.933978 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:25.935040 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.935555 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.936832 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.937046 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.937134 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:25.940160 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:25.940167 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:25.940170 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:25.940204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:25.941939 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.942434 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.942441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.942445 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.942539 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:25.942974 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:25.942980 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:25.942985 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:25.943244 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:25.943360 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:25.943836 (+ 476us) server_negotiation.cc:299] Negotiation successful
0811 04:37:25.943969 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:37:25.944803 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:25.931691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41117 (local address 127.2.238.130:45177)
0811 04:37:25.932149 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:25.932190 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:25.933583 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:25.934144 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:25.934150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:25.934468 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:25.934902 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:25.934914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.935692 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.935698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:25.936703 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:25.936715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:25.937195 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:25.937199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:25.937374 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:25.938013 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:25.938028 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:25.939973 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:25.942055 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:25.942059 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:25.942070 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:25.942331 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:25.942657 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:25.942661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:25.942663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:25.942866 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:25.943351 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:25.943355 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:25.943566 (+ 211us) client_negotiation.cc:769] Sending connection context
0811 04:37:25.943747 (+ 181us) client_negotiation.cc:240] Negotiation successful
0811 04:37:25.943990 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260811 04:37:25.946007 14419 heartbeater.cc:347] Connected to a master server at 127.2.238.190:41117
I20260811 04:37:25.946264 14419 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:25.946847 14419 heartbeater.cc:511] Master 127.2.238.190:41117 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:37:25.948009 14066 ts_manager.cc:194] Registered new tserver with Master: 2d562c7730f54c59a058ec778ed26415 (127.2.238.130:39097)
I20260811 04:37:25.948539 14066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:45177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:37:25.964576 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:37:26.074801 14424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:37:26.075042 14424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:37:26.075081 14424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:37:26.078914 14424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:37:26.078961 14424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:37:26.079056 14424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:37:26.084144 14424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:41117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:37:26.085122 14424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:37:26.086062 14424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:37:26.095477 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:37:26.095521 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:26.095721 14424 server_base.cc:1034] running on GCE node
W20260811 04:37:26.095795 14429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:37:26.096163 14424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:37:26.096740 14424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:37:26.097916 14424 hybrid_clock.cc:648] HybridClock initialized: now 1786423046097886 us; error 42 us; skew 500 ppm
Aug 11 04:37:26 dist-test-slave-0kls krb5kdc[14016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:37:26.100485 14424 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:37:26.101338 14424 webserver.cc:460] Webserver started at http://127.2.238.131:40281/ using document root <none> and password file <none>
I20260811 04:37:26.101945 14424 fs_manager.cc:359] Metadata directory not provided
I20260811 04:37:26.102036 14424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:37:26.102283 14424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:37:26.104306 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "49f074e68da440c697848bd6e2ba0022"
format_stamp: "Formatted at 2026-08-11 04:37:26 on dist-test-slave-0kls"
server_key: "237bffa7087f8235c19ee680400e63ee"
server_key_iv: "e5fe8e2ef857afe844ddbf30e5d979a9"
server_key_version: "encryptionkey@0"
I20260811 04:37:26.104889 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "49f074e68da440c697848bd6e2ba0022"
format_stamp: "Formatted at 2026-08-11 04:37:26 on dist-test-slave-0kls"
server_key: "237bffa7087f8235c19ee680400e63ee"
server_key_iv: "e5fe8e2ef857afe844ddbf30e5d979a9"
server_key_version: "encryptionkey@0"
I20260811 04:37:26.108364 14424 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260811 04:37:26.110762 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:26.112027 14424 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:37:26.112206 14424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "49f074e68da440c697848bd6e2ba0022"
format_stamp: "Formatted at 2026-08-11 04:37:26 on dist-test-slave-0kls"
server_key: "237bffa7087f8235c19ee680400e63ee"
server_key_iv: "e5fe8e2ef857afe844ddbf30e5d979a9"
server_key_version: "encryptionkey@0"
I20260811 04:37:26.112321 14424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:37:26.134232 14424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:37:26.137138 14424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:37:26.137363 14424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:37:26.138003 14424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:37:26.139006 14424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:37:26.139055 14424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:26.139102 14424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:37:26.139124 14424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:37:26.147743 14424 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:46119
I20260811 04:37:26.147765 14560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:46119 every 8 connection(s)
I20260811 04:37:26.148571 14424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:37:26.150725 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 14424
I20260811 04:37:26.150851 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:37:26.151243 3002 external_mini_cluster.cc:1428] Setting key 0951d58d2255a81febb4ccaa6a2449c4
Aug 11 04:37:26 dist-test-slave-0kls krb5kdc[14016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423046, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:37:26.162791 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.150374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:45791 (local address 127.2.238.190:41117)
0811 04:37:26.150544 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.150551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.151503 (+ 952us) server_negotiation.cc:407] Connection header received
0811 04:37:26.153251 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.153253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.153310 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.153390 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:26.154757 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.155285 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.156353 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.156558 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.156667 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.159065 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:26.159068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:26.159070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:26.159095 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:26.160523 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:26.160974 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:26.160976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:26.160978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:26.161065 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:26.161346 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:26.161348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:26.161349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:26.161638 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:26.161741 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.162467 (+ 726us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.162654 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:37:26.163581 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.150744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41117 (local address 127.2.238.131:45791)
0811 04:37:26.151336 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.151396 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.153039 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.153612 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.153620 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.154150 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:37:26.154620 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.154633 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.155405 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.155408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.156233 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.156241 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.156704 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.156707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.156876 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.157410 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:37:26.157426 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:37:26.158885 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:37:26.160631 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:26.160636 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:26.160648 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:26.160885 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:26.161140 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:37:26.161142 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:37:26.161143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:37:26.161262 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:37:26.161774 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:37:26.161812 (+ 38us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:37:26.162167 (+ 355us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.162412 (+ 245us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.162748 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":121,"threads_started":1}
I20260811 04:37:26.164803 14562 heartbeater.cc:347] Connected to a master server at 127.2.238.190:41117
I20260811 04:37:26.165060 14562 heartbeater.cc:464] Registering TS with master...
I20260811 04:37:26.165596 14562 heartbeater.cc:511] Master 127.2.238.190:41117 requested a full tablet report, sending...
I20260811 04:37:26.166615 14066 ts_manager.cc:194] Registered new tserver with Master: 49f074e68da440c697848bd6e2ba0022 (127.2.238.131:46119)
I20260811 04:37:26.167116 14066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:45791
I20260811 04:37:26.177695 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:37:26.188655 14110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.179848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.2.238.190:41117)
0811 04:37:26.180080 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.180085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.180212 (+ 127us) server_negotiation.cc:407] Connection header received
0811 04:37:26.180369 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.180373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.180421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.180514 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:37:26.181284 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.181859 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.183116 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.183539 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.183649 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.185296 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:37:26.185300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:37:26.185302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:37:26.185329 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:37:26.186741 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:26.187197 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:26.187201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:26.187203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:26.187299 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:37:26.187644 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:37:26.187648 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:37:26.187650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:37:26.187954 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:37:26.188106 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.188348 (+ 242us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.188473 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260811 04:37:26.191608 14066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:37:26.194941 14066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:37:26.209511 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.204129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:39097 (local address 127.0.0.1:53934)
0811 04:37:26.204355 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.204366 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.204459 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.205137 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.205144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.205165 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:26.205317 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.205323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.206972 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.206975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.208089 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.208099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.208917 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.208920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.209026 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209061 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.209194 (+ 133us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.209335 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":18,"thread_start_us":99,"threads_started":1}
I20260811 04:37:26.209491 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.204269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53934 (local address 127.2.238.130:39097)
0811 04:37:26.204618 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.204629 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.204650 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:37:26.204725 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.204730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.204973 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.205487 (+ 514us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:26.205651 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.206834 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.208208 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.208787 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.208915 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209064 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.209225 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.209324 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":117,"threads_started":1}
I20260811 04:37:26.209540 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.203410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33793 (local address 127.0.0.1:49968)
0811 04:37:26.203886 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.203934 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.204060 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.205255 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.205258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.205275 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:26.205390 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.205395 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.206619 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.206624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.207945 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.207955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.209121 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.209124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.209216 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209238 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.209290 (+ 52us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.209358 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":27,"thread_start_us":61,"threads_started":1}
I20260811 04:37:26.209895 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.203677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.2.238.131:46119)
0811 04:37:26.204177 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.204188 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.205029 (+ 841us) server_negotiation.cc:407] Connection header received
0811 04:37:26.205229 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.205234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.205404 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.205527 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:26.205993 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.207019 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.208276 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.209011 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.209108 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209243 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.209600 (+ 357us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.209725 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":192,"threads_started":1}
I20260811 04:37:26.210265 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.203415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:46119 (local address 127.0.0.1:46872)
0811 04:37:26.204933 (+ 1518us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.204947 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.205101 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.205712 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.205714 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.205731 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:26.205855 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.205863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.207144 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.207146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.208132 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.208163 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.209301 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.209304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.209417 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209440 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.210081 (+ 641us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.210146 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1433,"mutex_wait_us":1238,"thread_start_us":92,"threads_started":1}
I20260811 04:37:26.210276 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.204431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49968 (local address 127.2.238.129:33793)
0811 04:37:26.204758 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.204774 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.204799 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:37:26.204893 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.204898 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.205098 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.205536 (+ 438us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:26.205742 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.206481 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.208114 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.208958 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.209795 (+ 837us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.209969 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.210023 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.210133 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260811 04:37:26.212839 14348 tablet_service.cc:1467] Processing CreateTablet for tablet c202391ca3fb41fc8d0af177ccb46142 (DEFAULT_TABLE table=test-table [id=f6fef89affa141c69d9f8f8ccb1a201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:26.212845 14204 tablet_service.cc:1467] Processing CreateTablet for tablet c202391ca3fb41fc8d0af177ccb46142 (DEFAULT_TABLE table=test-table [id=f6fef89affa141c69d9f8f8ccb1a201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:26.213855 14348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202391ca3fb41fc8d0af177ccb46142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:26.213855 14204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202391ca3fb41fc8d0af177ccb46142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:26.216748 14492 tablet_service.cc:1467] Processing CreateTablet for tablet c202391ca3fb41fc8d0af177ccb46142 (DEFAULT_TABLE table=test-table [id=f6fef89affa141c69d9f8f8ccb1a201f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:37:26.217829 14492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c202391ca3fb41fc8d0af177ccb46142. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:37:26.220606 14580 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: Bootstrap starting.
I20260811 04:37:26.221809 14581 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Bootstrap starting.
I20260811 04:37:26.222857 14580 tablet_bootstrap.cc:654] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:26.224002 14580 log.cc:824] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:26.224397 14581 tablet_bootstrap.cc:654] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:26.225435 14581 log.cc:824] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:26.225476 14583 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Bootstrap starting.
I20260811 04:37:26.226326 14580 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: No bootstrap required, opened a new log
I20260811 04:37:26.226549 14580 ts_tablet_manager.cc:1397] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:37:26.227285 14581 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: No bootstrap required, opened a new log
I20260811 04:37:26.227489 14581 ts_tablet_manager.cc:1397] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260811 04:37:26.228183 14583 tablet_bootstrap.cc:654] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Neither blocks nor log segments found. Creating new log.
I20260811 04:37:26.228971 14583 log.cc:824] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Log is configured to *not* fsync() on all Append() calls
I20260811 04:37:26.230971 14581 raft_consensus.cc:348] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.230993 14580 raft_consensus.cc:348] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.231215 14581 raft_consensus.cc:374] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:26.231258 14580 raft_consensus.cc:374] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:26.231264 14581 raft_consensus.cc:729] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d562c7730f54c59a058ec778ed26415, State: Initialized, Role: FOLLOWER
I20260811 04:37:26.231323 14580 raft_consensus.cc:729] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0020bf3894e54834a1666b711feb3ed8, State: Initialized, Role: FOLLOWER
I20260811 04:37:26.231735 14581 consensus_queue.cc:260] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.231907 14583 tablet_bootstrap.cc:492] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: No bootstrap required, opened a new log
I20260811 04:37:26.231808 14580 consensus_queue.cc:260] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.232095 14583 ts_tablet_manager.cc:1397] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260811 04:37:26.232810 14581 ts_tablet_manager.cc:1428] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260811 04:37:26.232808 14580 ts_tablet_manager.cc:1428] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:37:26.233197 14419 heartbeater.cc:503] Master 127.2.238.190:41117 was elected leader, sending a full tablet report...
I20260811 04:37:26.233973 14275 heartbeater.cc:503] Master 127.2.238.190:41117 was elected leader, sending a full tablet report...
I20260811 04:37:26.236366 14583 raft_consensus.cc:348] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.236644 14583 raft_consensus.cc:374] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:37:26.236712 14583 raft_consensus.cc:729] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49f074e68da440c697848bd6e2ba0022, State: Initialized, Role: FOLLOWER
I20260811 04:37:26.237183 14583 consensus_queue.cc:260] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.237980 14562 heartbeater.cc:503] Master 127.2.238.190:41117 was elected leader, sending a full tablet report...
I20260811 04:37:26.238332 14583 ts_tablet_manager.cc:1428] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
W20260811 04:37:26.399544 14563 tablet.cc:2369] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:37:26.412501 14587 raft_consensus.cc:482] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:37:26.412756 14587 raft_consensus.cc:504] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.414065 14587 leader_election.cc:288] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0020bf3894e54834a1666b711feb3ed8 (127.2.238.129:33793), 49f074e68da440c697848bd6e2ba0022 (127.2.238.131:46119)
I20260811 04:37:26.418792 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.414654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:48429 (local address 127.2.238.131:46119)
0811 04:37:26.414791 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.414796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.414899 (+ 103us) server_negotiation.cc:407] Connection header received
0811 04:37:26.415016 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.415022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.415065 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.415142 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:26.415546 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.416281 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.417785 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.418218 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.418358 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.418411 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.418600 (+ 189us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.418669 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:37:26.418828 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.414318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:46119 (local address 127.2.238.130:48429)
0811 04:37:26.414785 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.414799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.414917 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.415196 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.415199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.415220 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:26.415348 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.415392 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.416383 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.416386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.417663 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.417671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.418358 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.418362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.418457 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.418487 (+ 30us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.418570 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.418648 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":25,"thread_start_us":93,"threads_started":1}
I20260811 04:37:26.419072 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.414590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:33097 (local address 127.2.238.129:33793)
0811 04:37:26.414761 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.414766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.414780 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:37:26.414847 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.414849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.414945 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.415033 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:37:26.415276 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.415950 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.417607 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.418239 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.418357 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.418412 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.418886 (+ 474us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.418960 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260811 04:37:26.419327 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.414299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33793 (local address 127.2.238.130:33097)
0811 04:37:26.414464 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.414480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:37:26.414578 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:37:26.415045 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:37:26.415048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:37:26.415063 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:37:26.415174 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.415179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.416056 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.416061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.417453 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:37:26.417464 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.418470 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.418473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:37:26.418602 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.418624 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 04:37:26.418693 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 04:37:26.418781 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72,"mutex_wait_us":23}
I20260811 04:37:26.419796 14513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202391ca3fb41fc8d0af177ccb46142" candidate_uuid: "2d562c7730f54c59a058ec778ed26415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f074e68da440c697848bd6e2ba0022" is_pre_election: true
I20260811 04:37:26.419904 14226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202391ca3fb41fc8d0af177ccb46142" candidate_uuid: "2d562c7730f54c59a058ec778ed26415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0020bf3894e54834a1666b711feb3ed8" is_pre_election: true
I20260811 04:37:26.420161 14513 raft_consensus.cc:2454] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d562c7730f54c59a058ec778ed26415 in term 0.
I20260811 04:37:26.420161 14226 raft_consensus.cc:2454] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d562c7730f54c59a058ec778ed26415 in term 0.
I20260811 04:37:26.420619 14298 leader_election.cc:302] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d562c7730f54c59a058ec778ed26415, 49f074e68da440c697848bd6e2ba0022; no voters:
I20260811 04:37:26.420902 14587 raft_consensus.cc:2781] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:37:26.421010 14587 raft_consensus.cc:482] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:37:26.421054 14587 raft_consensus.cc:3037] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:26.422324 14587 raft_consensus.cc:504] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.422672 14587 leader_election.cc:288] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [CANDIDATE]: Term 1 election: Requested vote from peers 0020bf3894e54834a1666b711feb3ed8 (127.2.238.129:33793), 49f074e68da440c697848bd6e2ba0022 (127.2.238.131:46119)
I20260811 04:37:26.423071 14226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202391ca3fb41fc8d0af177ccb46142" candidate_uuid: "2d562c7730f54c59a058ec778ed26415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0020bf3894e54834a1666b711feb3ed8"
I20260811 04:37:26.423087 14513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c202391ca3fb41fc8d0af177ccb46142" candidate_uuid: "2d562c7730f54c59a058ec778ed26415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49f074e68da440c697848bd6e2ba0022"
I20260811 04:37:26.423231 14513 raft_consensus.cc:3037] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:26.423247 14226 raft_consensus.cc:3037] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:37:26.424350 14226 raft_consensus.cc:2454] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d562c7730f54c59a058ec778ed26415 in term 1.
I20260811 04:37:26.424408 14513 raft_consensus.cc:2454] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d562c7730f54c59a058ec778ed26415 in term 1.
I20260811 04:37:26.424691 14299 leader_election.cc:302] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0020bf3894e54834a1666b711feb3ed8, 2d562c7730f54c59a058ec778ed26415; no voters:
I20260811 04:37:26.424914 14587 raft_consensus.cc:2781] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:37:26.425177 14587 raft_consensus.cc:686] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 1 LEADER]: Becoming Leader. State: Replica: 2d562c7730f54c59a058ec778ed26415, State: Running, Role: LEADER
I20260811 04:37:26.425525 14587 consensus_queue.cc:237] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } }
I20260811 04:37:26.428717 14066 catalog_manager.cc:5495] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d562c7730f54c59a058ec778ed26415 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "2d562c7730f54c59a058ec778ed26415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d562c7730f54c59a058ec778ed26415" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 39097 } health_report { overall_health: HEALTHY } } }
W20260811 04:37:26.430738 14420 tablet.cc:2369] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:37:26.444090 14577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.440820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.2.238.130:39097)
0811 04:37:26.440935 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:37:26.440938 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:37:26.440964 (+ 26us) server_negotiation.cc:407] Connection header received
0811 04:37:26.441113 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:37:26.441116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:37:26.441167 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:37:26.441254 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:37:26.441476 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.442054 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.442856 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:37:26.443054 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:37:26.443146 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:37:26.443423 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:37:26.443795 (+ 372us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:37:26.443881 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0811 04:37:26.443933 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 04:37:26.443977 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:37:26.449715 14513 raft_consensus.cc:1261] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 1 FOLLOWER]: Refusing update from remote peer 2d562c7730f54c59a058ec778ed26415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:26.449726 14226 raft_consensus.cc:1261] T c202391ca3fb41fc8d0af177ccb46142 P 0020bf3894e54834a1666b711feb3ed8 [term 1 FOLLOWER]: Refusing update from remote peer 2d562c7730f54c59a058ec778ed26415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:37:26.450519 14591 consensus_queue.cc:1035] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49f074e68da440c697848bd6e2ba0022" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 46119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:26.450693 14587 consensus_queue.cc:1035] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0020bf3894e54834a1666b711feb3ed8" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:37:26.458371 14595 mvcc.cc:205] Tried to move back new op lower bound from 7317188798251216896 to 7317188798161707008. Current Snapshot: MvccSnapshot[applied={T|T < 7317188798251216896}]
I20260811 04:37:26.467288 14064 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:26.467446 14064 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:37:26.469980 14064 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 41eecedc1c2449e08b546e2b858301bc: Sending DeleteTablet for 3 replicas of tablet c202391ca3fb41fc8d0af177ccb46142
I20260811 04:37:26.470996 14492 tablet_service.cc:1514] Processing DeleteTablet for tablet c202391ca3fb41fc8d0af177ccb46142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:26 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:46872
I20260811 04:37:26.471051 14348 tablet_service.cc:1514] Processing DeleteTablet for tablet c202391ca3fb41fc8d0af177ccb46142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:26 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:53934
I20260811 04:37:26.471267 14204 tablet_service.cc:1514] Processing DeleteTablet for tablet c202391ca3fb41fc8d0af177ccb46142 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:37:26 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49968
I20260811 04:37:26.471547 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 14135
I20260811 04:37:26.472216 14605 tablet_replica.cc:330] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: stopping tablet replica
I20260811 04:37:26.472640 14605 raft_consensus.cc:2229] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:37:26.472995 14605 raft_consensus.cc:2258] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:26.475059 14606 tablet_replica.cc:330] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: stopping tablet replica
I20260811 04:37:26.475380 14606 raft_consensus.cc:2229] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:37:26.475798 14606 raft_consensus.cc:2258] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:37:26.477620 14605 ts_tablet_manager.cc:1905] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:26.477620 14606 ts_tablet_manager.cc:1905] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:37:26.480648 14605 ts_tablet_manager.cc:1918] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:26.480752 14605 log.cc:1196] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/c202391ca3fb41fc8d0af177ccb46142
I20260811 04:37:26.481104 14605 ts_tablet_manager.cc:1939] T c202391ca3fb41fc8d0af177ccb46142 P 49f074e68da440c697848bd6e2ba0022: Deleting consensus metadata
I20260811 04:37:26.481532 14606 ts_tablet_manager.cc:1918] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:37:26.481704 14606 log.cc:1196] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/c202391ca3fb41fc8d0af177ccb46142
I20260811 04:37:26.482151 14606 ts_tablet_manager.cc:1939] T c202391ca3fb41fc8d0af177ccb46142 P 2d562c7730f54c59a058ec778ed26415: Deleting consensus metadata
I20260811 04:37:26.482498 14050 catalog_manager.cc:4841] TS 49f074e68da440c697848bd6e2ba0022 (127.2.238.131:46119): tablet c202391ca3fb41fc8d0af177ccb46142 (table test-table [id=f6fef89affa141c69d9f8f8ccb1a201f]) successfully deleted
W20260811 04:37:26.483161 14051 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:33793 (error 108)
I20260811 04:37:26.483669 14051 catalog_manager.cc:4841] TS 2d562c7730f54c59a058ec778ed26415 (127.2.238.130:39097): tablet c202391ca3fb41fc8d0af177ccb46142 (table test-table [id=f6fef89affa141c69d9f8f8ccb1a201f]) successfully deleted
I20260811 04:37:26.484258 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:37:26.483985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33793 (local address 127.0.0.1:49974)
0811 04:37:26.484113 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 04:37:26.484186 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 04:37:26.484505 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 14280
W20260811 04:37:26.486985 14051 catalog_manager.cc:4584] TS 0020bf3894e54834a1666b711feb3ed8 (127.2.238.129:33793): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c202391ca3fb41fc8d0af177ccb46142: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33793: connect: Connection refused (error 111)
I20260811 04:37:26.493748 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 14424
I20260811 04:37:26.501211 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 14030
2026-08-11T04:37:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2430 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:37:26 dist-test-slave-0kls krb5kdc[14611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:37:26 dist-test-slave-0kls krb5kdc[14611](info): set up 2 sockets
Aug 11 04:37:26 dist-test-slave-0kls krb5kdc[14611](info): commencing operation
krb5kdc: starting...
W20260811 04:37:27.591799 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:37:27 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423047, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:37:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangeradmin_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangeradmin_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangerlookup_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangerlookup_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
I20260811 04:37:27.762830 3002 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taski3hQIv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/postgres -l logfile start
2026-08-11 04:37:30.227 UTC [14643] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 04:37:30.227 UTC [14643] LOG: listening on IPv4 address "127.2.238.148", port 58061
2026-08-11 04:37:30.231 UTC [14643] LOG: listening on Unix socket "/tmp/.s.PGSQL.58061"
2026-08-11 04:37:30.253 UTC [14646] LOG: database system was shut down at 2026-08-11 04:37:28 UTC
2026-08-11 04:37:30.257 UTC [14643] LOG: database system is ready to accept connections
I20260811 04:37:31.235394 3002 mini_postgres.cc:96] Postgres bound to 58061
2026-08-11 04:37:31.243 UTC [14654] FATAL: database "slave" does not exist
127.2.238.148:58061 - accepting connections
I20260811 04:37:31.244300 3002 mini_ranger.cc:161] Starting Ranger...
I20260811 04:37:31.261487 3002 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 04:37:31.748294 3002 mini_ranger.cc:87] Created ranger Postgres database
I20260811 04:37:31.748400 3002 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin
2026-08-11 04:37:32,107 [I] DB FLAVOR :POSTGRES
2026-08-11 04:37:32,108 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 04:37:32,108 [I] Checking connection..
2026-08-11 04:37:32,108 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 04:37:32,378 [I] Checking connection passed.
2026-08-11 04:37:32,379 [I] --------- Verifying version history table ---------
2026-08-11 04:37:32,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:37:32,625 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 04:37:32,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:37:32,887 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 04:37:32,887 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 04:37:32,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 04:37:33.140 UTC [14720] WARNING: there is no transaction in progress
2026-08-11 04:37:33,147 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 04:37:33,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:37:33,390 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 04:37:33,390 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 04:37:33,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 04:37:33,637 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 04:37:33,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0kls.c.gcp-upstream.internal','N') ;"
2026-08-11 04:37:34,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 04:37:34,403 [I] Table x_portal_user does not exist in database ranger
2026-08-11 04:37:34,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 04:37:34,646 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 04:37:34,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 04:37:34,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 04:37:35,141 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 04:37:35,141 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 04:37:36.121 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.137 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.154 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.169 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.186 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.204 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.263 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.270 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.285 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.296 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.306 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.320 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.329 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.336 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.342 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36.349 UTC [14836] WARNING: there is no transaction in progress
2026-08-11 04:37:36,979 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 04:37:36,979 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 04:37:37,229 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 04:37:37,229 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 04:37:37,478 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 04:37:37,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 04:37:37,737 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 04:37:37,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 04:37:37,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 04:37:38,239 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0kls.c.gcp-upstream.internal';"
2026-08-11 04:37:38,478 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 04:37:38,478 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 04:37:38,738 [I] DB_PATCHES have already been applied
I20260811 04:37:38.743994 3002 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taski3hQIv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 04:37:38.744073 3002 mini_ranger.cc:193] Using host: 127.2.238.148
I20260811 04:37:38.748623 3002 mini_ranger.cc:227] Ranger admin URL: http://127.2.238.148:38529
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 4:37:39 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 04:37:39.901554 3002 mini_ranger.cc:160] Time spent starting Ranger: real 8.657s user 0.000s sys 0.013s
Aug 11, 2026 4:37:58 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 4:37:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 4:37:58 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 4:37:58 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 4:37:59 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 04:38:00.375056 3002 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangerkms_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/rangerkms_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.238.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.238.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 04:38:00 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423080, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 04:38:00.469585 3002 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 04:38:00.480180 3002 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 04:38:01.341079 3002 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 04:38:01.341171 3002 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms
I20260811 04:38:01.341290 3002 mini_ranger_kms.cc:227] Using postgres at 127.2.238.148:58061
2026-08-11 04:38:01,568 [I] DB FLAVOR :POSTGRES
2026-08-11 04:38:01,568 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 04:38:01,568 [I] Checking connection
2026-08-11 04:38:01,569 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 04:38:01,802 [I] connection success
2026-08-11 04:38:01,803 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 04:38:01,803 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 04:38:01,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 04:38:02,060 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 04:38:02,061 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 04:38:02,061 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 04:38:02,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:58061/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 04:38:02,331 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 04:38:02.803458 3002 mini_ranger_kms.cc:310] Created kms service
I20260811 04:38:02.993299 3002 mini_ranger_kms.cc:326] Created kudu user
I20260811 04:38:03.035811 3002 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 04:38:03 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423080, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:03.401466 3002 mini_ranger_kms.cc:384] Added ranger policy
I20260811 04:38:03.401638 3002 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms:/tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 04:38:03.401678 3002 mini_ranger_kms.cc:240] Using host: 127.2.238.148
I20260811 04:38:03.406538 3002 mini_ranger_kms.cc:276] Ranger KMS PID: 15025
I20260811 04:38:03.406677 3002 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.238.148:57259
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 4:38:04 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 04:38:05 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786423085, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:05.659430 3002 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.190s user 0.023s sys 0.188s
I20260811 04:38:05.659556 3002 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 04:38:05.659657 3002 mini_ranger_kms.cc:401] 127.2.238.148:57259/kms/v1/keys
Aug 11 04:38:05 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423085, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
Aug 11 04:38:06 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423085, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
Aug 11 04:38:08 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:38:08.559441 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36281
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:36281
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:57259/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:08.687732 15079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:08.687965 15079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:08.688000 15079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:08.691846 15079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:38:08.691906 15079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:08.691923 15079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:08.691938 15079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:38:08.691952 15079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:38:08.697212 15079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.238.148:57259/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:36281
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:08.698299 15079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:08.699401 15079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:08.708089 15084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:08.708096 15085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:08.708108 15087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:08.708325 15079 server_base.cc:1034] running on GCE node
I20260811 04:38:08.709144 15079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:08.710373 15079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:08.711546 15079 hybrid_clock.cc:648] HybridClock initialized: now 1786423088711524 us; error 37 us; skew 500 ppm
Aug 11 04:38:08 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423088, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:08.714620 15079 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:38:08.715600 15079 webserver.cc:460] Webserver started at http://127.2.238.190:41907/ using document root <none> and password file <none>
I20260811 04:38:08.716183 15079 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:08.716262 15079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:08.716496 15079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:08 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423088, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:08.777866 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "db973d359e87484dbe1a66a953586ba7"
format_stamp: "Formatted at 2026-08-11 04:38:08 on dist-test-slave-0kls"
server_key: "f5ae7eb233cd529564fe499979313fe3"
server_key_iv: "f7491d87bb68448616648cc6c742163d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:08.778607 15079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "db973d359e87484dbe1a66a953586ba7"
format_stamp: "Formatted at 2026-08-11 04:38:08 on dist-test-slave-0kls"
server_key: "f5ae7eb233cd529564fe499979313fe3"
server_key_iv: "f7491d87bb68448616648cc6c742163d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:08.782883 15079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260811 04:38:08.800132 15094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:08.801841 15079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:38:08.801981 15079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "db973d359e87484dbe1a66a953586ba7"
format_stamp: "Formatted at 2026-08-11 04:38:08 on dist-test-slave-0kls"
server_key: "f5ae7eb233cd529564fe499979313fe3"
server_key_iv: "f7491d87bb68448616648cc6c742163d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:08.802125 15079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:08.833097 15079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:08.836364 15079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:08.836545 15079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:08.844726 15079 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:36281
I20260811 04:38:08.844722 15148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:36281 every 8 connection(s)
I20260811 04:38:08.846015 15079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:38:08.848723 15150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:08.849053 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 15079
I20260811 04:38:08.849179 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:38:08.849458 3002 external_mini_cluster.cc:1428] Setting key df84549819e778bf4ed463b3531b15c9
I20260811 04:38:08.855192 15150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: Bootstrap starting.
I20260811 04:38:08.857757 15150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:08.858790 15150 log.cc:824] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:38:08 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:08.861604 15150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: No bootstrap required, opened a new log
I20260811 04:38:08.865623 15150 raft_consensus.cc:348] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } }
I20260811 04:38:08.866075 15150 raft_consensus.cc:374] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:08.866151 15150 raft_consensus.cc:729] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db973d359e87484dbe1a66a953586ba7, State: Initialized, Role: FOLLOWER
I20260811 04:38:08.866675 15150 consensus_queue.cc:260] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } }
I20260811 04:38:08.866806 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:08.851333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50912 (local address 127.2.238.190:36281)
0811 04:38:08.851774 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:08.851791 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:08.851831 (+ 40us) server_negotiation.cc:407] Connection header received
0811 04:38:08.852633 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:08.852654 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:08.853040 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:08.853394 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:08.854260 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:08.855785 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:08.856971 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:08.857644 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:08.857813 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:08.860682 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:08.860689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:08.860706 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:08.860735 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:08.863369 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:08.864011 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:08.864017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:08.864023 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:08.864166 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:08.864468 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:08.864471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:08.864473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:08.864971 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:08.865172 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:08.865466 (+ 294us) server_negotiation.cc:299] Negotiation successful
0811 04:38:08.865852 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":147,"threads_started":1}
I20260811 04:38:08.866830 15150 raft_consensus.cc:388] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:38:08.867090 15150 raft_consensus.cc:482] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:38:08.867233 15150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:08.868343 15150 raft_consensus.cc:504] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } }
I20260811 04:38:08.868683 15150 leader_election.cc:302] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: db973d359e87484dbe1a66a953586ba7; no voters:
I20260811 04:38:08.869025 15150 leader_election.cc:288] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:38:08.869405 15157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:38:08.869664 15157 raft_consensus.cc:686] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [term 1 LEADER]: Becoming Leader. State: Replica: db973d359e87484dbe1a66a953586ba7, State: Running, Role: LEADER
I20260811 04:38:08.870012 15157 consensus_queue.cc:237] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } }
I20260811 04:38:08.870299 15150 sys_catalog.cc:564] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:38:08.871744 15158 sys_catalog.cc:455] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader db973d359e87484dbe1a66a953586ba7. Latest consensus state: current_term: 1 leader_uuid: "db973d359e87484dbe1a66a953586ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } } }
I20260811 04:38:08.871872 15158 sys_catalog.cc:458] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:08.872233 15156 sys_catalog.cc:455] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "db973d359e87484dbe1a66a953586ba7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "db973d359e87484dbe1a66a953586ba7" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36281 } } }
I20260811 04:38:08.872337 15156 sys_catalog.cc:458] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:08.872848 15167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:38:08.872911 15150 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 04:38:08.882153 15167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:38:08.890472 15167 catalog_manager.cc:1347] Generated new cluster ID: 4f9cebc5af824e37a6c5216367faaee3
I20260811 04:38:08.890532 15167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:38:08.901448 15167 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:38:08.902823 15167 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:38:08.909302 15167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: Generated new TSK 0
I20260811 04:38:08.910068 15167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:38:09.585736 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:57259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:09.703815 15202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:09.704126 15202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:09.704241 15202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:09.708606 15202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:09.708686 15202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:09.708793 15202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:38:09.714303 15202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.238.148:57259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:09.715443 15202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:09.716621 15202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:09.727530 15208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:09.727532 15211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:09.727530 15207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:09.728390 15202 server_base.cc:1034] running on GCE node
I20260811 04:38:09.728850 15202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:09.729514 15202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:09.730707 15202 hybrid_clock.cc:648] HybridClock initialized: now 1786423089730688 us; error 37 us; skew 500 ppm
Aug 11 04:38:09 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:09.733688 15202 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:38:09.734771 15202 webserver.cc:460] Webserver started at http://127.2.238.129:32941/ using document root <none> and password file <none>
I20260811 04:38:09.735394 15202 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:09.735463 15202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:09.735698 15202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:09 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:09.753326 15202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "f54c89c7e84643559069d0918dae7050"
format_stamp: "Formatted at 2026-08-11 04:38:09 on dist-test-slave-0kls"
server_key: "e5fe782b1533beeb0fb0c3bef7827edb"
server_key_iv: "d1d28a0965aa73841357aa0a169c0a1e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:09.754307 15202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "f54c89c7e84643559069d0918dae7050"
format_stamp: "Formatted at 2026-08-11 04:38:09 on dist-test-slave-0kls"
server_key: "e5fe782b1533beeb0fb0c3bef7827edb"
server_key_iv: "d1d28a0965aa73841357aa0a169c0a1e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:09.762789 15202 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260811 04:38:09.786047 15222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:09.787462 15202 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.004s sys 0.000s
I20260811 04:38:09.787614 15202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "f54c89c7e84643559069d0918dae7050"
format_stamp: "Formatted at 2026-08-11 04:38:09 on dist-test-slave-0kls"
server_key: "e5fe782b1533beeb0fb0c3bef7827edb"
server_key_iv: "d1d28a0965aa73841357aa0a169c0a1e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:09.787761 15202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:09.808471 15202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:09.812243 15202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:09.812486 15202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:09.813292 15202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:09.814410 15202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:09.814476 15202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:09.814532 15202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:09.814623 15202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260811 04:38:09.826347 15202 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:37543
I20260811 04:38:09.826527 15343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:37543 every 8 connection(s)
I20260811 04:38:09.827525 15202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:38:09.833398 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 15202
I20260811 04:38:09.833527 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:38:09.833922 3002 external_mini_cluster.cc:1428] Setting key cfd452013f1994c1259ae994dda854f1
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Aug 11 04:38:09 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:09.875233 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:09.842723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36281 (local address 127.2.238.129:59405)
0811 04:38:09.856477 (+ 13754us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:09.856558 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:09.857747 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:09.859635 (+ 1888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:09.859645 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:09.860032 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:09.860754 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:09.860774 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:09.861971 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:09.861975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:09.863368 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:09.863382 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:09.864226 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:09.864231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:09.864507 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:09.865485 (+ 978us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:09.865513 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:09.867340 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:09.871660 (+ 4320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:09.871667 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:09.871686 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:09.872074 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:09.872483 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:09.872487 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:09.872489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:09.872702 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:09.873216 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:09.873222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:09.873537 (+ 315us) client_negotiation.cc:769] Sending connection context
0811 04:38:09.873724 (+ 187us) client_negotiation.cc:240] Negotiation successful
0811 04:38:09.874117 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13463,"thread_start_us":165,"threads_started":1}
I20260811 04:38:09.876664 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:09.843956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:59405 (local address 127.2.238.190:36281)
0811 04:38:09.859051 (+ 15095us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:09.859060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:09.859077 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:38:09.859121 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:09.859124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:09.859177 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:09.859272 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:09.860932 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:09.861780 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:09.863526 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:09.863892 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:09.864006 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:09.868040 (+ 4034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:09.868046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:09.868052 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:09.868096 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:09.871506 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:09.872213 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:09.872216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:09.872218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:09.872370 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:09.872851 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:09.872854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:09.872856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:09.873091 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:09.876256 (+ 3165us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:09.876341 (+ 85us) server_negotiation.cc:299] Negotiation successful
0811 04:38:09.876491 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15007,"thread_start_us":107,"threads_started":1}
I20260811 04:38:09.877995 15344 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36281
I20260811 04:38:09.878243 15344 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:09.878875 15344 heartbeater.cc:511] Master 127.2.238.190:36281 requested a full tablet report, sending...
I20260811 04:38:09.880810 15111 ts_manager.cc:194] Registered new tserver with Master: f54c89c7e84643559069d0918dae7050 (127.2.238.129:37543)
I20260811 04:38:09.882412 15111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:59405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:38:09.886778 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:57259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:38:09 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 04:38:10.049317 15353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:10.049634 15353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:10.049701 15353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:10.055142 15353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:10.055272 15353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:10.055408 15353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:38:10.063397 15353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.238.148:57259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:10.064747 15353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:10.066159 15353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:10.078945 15359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:10.079217 15362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:10.079066 15360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:10.080379 15353 server_base.cc:1034] running on GCE node
I20260811 04:38:10.080875 15353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:10.081748 15353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:10.082975 15353 hybrid_clock.cc:648] HybridClock initialized: now 1786423090082954 us; error 36 us; skew 500 ppm
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:10.086650 15353 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:38:10.087633 15353 webserver.cc:460] Webserver started at http://127.2.238.130:44191/ using document root <none> and password file <none>
I20260811 04:38:10.088274 15353 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:10.088363 15353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:10.088595 15353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:10.106899 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "180bcb75123946f88e41861e388cbca0"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "d16419eecd9e5740b24ef8231a1e5609"
server_key_iv: "bd389849a5d964ec0c44683fa4cfc71e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.107820 15353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "180bcb75123946f88e41861e388cbca0"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "d16419eecd9e5740b24ef8231a1e5609"
server_key_iv: "bd389849a5d964ec0c44683fa4cfc71e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.113406 15353 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260811 04:38:10.130218 15370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.131954 15353 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260811 04:38:10.132071 15353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "180bcb75123946f88e41861e388cbca0"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "d16419eecd9e5740b24ef8231a1e5609"
server_key_iv: "bd389849a5d964ec0c44683fa4cfc71e"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.132186 15353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:10.148479 15353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:10.152328 15353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:10.152551 15353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:10.153285 15353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:10.154456 15353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:10.154561 15353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.154651 15353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:10.154711 15353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.165464 15353 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:37077
I20260811 04:38:10.166777 15353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:38:10.176474 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 15353
I20260811 04:38:10.176643 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:38:10.176990 3002 external_mini_cluster.cc:1428] Setting key fb4e33c4e7b47d6a9864d20930347c23
I20260811 04:38:10.181860 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:37077 every 8 connection(s)
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:38:10.203162 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.186270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:33393 (local address 127.2.238.190:36281)
0811 04:38:10.186416 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.186421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.188462 (+ 2041us) server_negotiation.cc:407] Connection header received
0811 04:38:10.189744 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.189748 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.189834 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.189891 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:10.191912 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.192795 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.194645 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.195013 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.195093 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.198662 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:10.198666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:10.198668 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:10.198693 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:10.200517 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.201178 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.201182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.201183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.201288 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.201715 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.201719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.201720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.201985 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:10.202081 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.202824 (+ 743us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.202962 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:38:10.205175 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.185089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36281 (local address 127.2.238.130:33393)
0811 04:38:10.188304 (+ 3215us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.188363 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.189497 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.190342 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.190351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.190861 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:10.191720 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.191743 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.193052 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.193056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.194495 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.194507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.195328 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.195332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.195528 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.196664 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:10.196685 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:10.198470 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:10.200668 (+ 2198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:10.200675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:10.200692 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:10.201043 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:10.201409 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:10.201412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:10.201414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:10.201590 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:10.202159 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:10.202166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:10.202521 (+ 355us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.203697 (+ 1176us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.204075 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2946,"thread_start_us":191,"threads_started":1}
I20260811 04:38:10.206986 15489 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36281
I20260811 04:38:10.207252 15489 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:10.207926 15489 heartbeater.cc:511] Master 127.2.238.190:36281 requested a full tablet report, sending...
I20260811 04:38:10.209345 15111 ts_manager.cc:194] Registered new tserver with Master: 180bcb75123946f88e41861e388cbca0 (127.2.238.130:37077)
I20260811 04:38:10.210150 15111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:33393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:38:10.240882 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:57259/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:10.413582 15494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:10.413985 15494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:10.414088 15494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:10.420106 15494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:10.420252 15494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:10.420382 15494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:38:10.427597 15494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.238.148:57259/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:36281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:10.429129 15494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:10.430562 15494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:10.443442 15503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:10.444466 15501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:10.444772 15494 server_base.cc:1034] running on GCE node
W20260811 04:38:10.447515 15500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:10.448097 15494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:10.448870 15494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:10.450011 15494 hybrid_clock.cc:648] HybridClock initialized: now 1786423090449942 us; error 86 us; skew 500 ppm
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:10.453788 15494 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:38:10.454921 15494 webserver.cc:460] Webserver started at http://127.2.238.131:35405/ using document root <none> and password file <none>
I20260811 04:38:10.455679 15494 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:10.455816 15494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:10.456112 15494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:10.476274 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "0eecec8c32644f80aac914c2ca5163ee"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "efcf56a15bb198c68f338eed591527ae"
server_key_iv: "df7817570ced9c81465e3e831f2cd916"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.477728 15494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "0eecec8c32644f80aac914c2ca5163ee"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "efcf56a15bb198c68f338eed591527ae"
server_key_iv: "df7817570ced9c81465e3e831f2cd916"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.484499 15494 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:10.517318 15510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.519291 15494 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.000s sys 0.003s
I20260811 04:38:10.519435 15494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "0eecec8c32644f80aac914c2ca5163ee"
format_stamp: "Formatted at 2026-08-11 04:38:10 on dist-test-slave-0kls"
server_key: "efcf56a15bb198c68f338eed591527ae"
server_key_iv: "df7817570ced9c81465e3e831f2cd916"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:10.519582 15494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:10.545717 15494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:10.552304 15494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:10.552560 15494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:10.553356 15494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:10.554512 15494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:10.554610 15494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.554710 15494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:10.554770 15494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:10.565979 15494 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:45007
I20260811 04:38:10.567094 15494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:38:10.571381 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 15494
I20260811 04:38:10.571492 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:38:10.571759 3002 external_mini_cluster.cc:1428] Setting key c5e57c8b719bb2eca519a4c7733f0d84
I20260811 04:38:10.577991 15628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:45007 every 8 connection(s)
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423090, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:10.594918 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.577718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:55243 (local address 127.2.238.190:36281)
0811 04:38:10.577858 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.577863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.578539 (+ 676us) server_negotiation.cc:407] Connection header received
0811 04:38:10.579664 (+ 1125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.579667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.579718 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.579765 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:10.581390 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.582232 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.583827 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.584094 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.584194 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.590390 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:10.590394 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:10.590398 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:10.590427 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:10.592275 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.592964 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.592968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.592970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.593075 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.593494 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.593498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.593499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.593757 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:10.593871 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.594611 (+ 740us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.594743 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 04:38:10.596764 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.576573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:36281 (local address 127.2.238.131:55243)
0811 04:38:10.578393 (+ 1820us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.578444 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.579440 (+ 996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.580204 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.580213 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.580554 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:10.581237 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.581254 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.582384 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.582387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.583696 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.583707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.584417 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.584422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.584603 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.587121 (+ 2518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:10.587141 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:10.590155 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:10.592414 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:10.592420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:10.592438 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:10.592841 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:10.593197 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:10.593200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:10.593202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:10.593384 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:10.593959 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:10.593966 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:10.594301 (+ 335us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.595327 (+ 1026us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.595706 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"thread_start_us":236,"threads_started":1}
I20260811 04:38:10.598291 15629 heartbeater.cc:347] Connected to a master server at 127.2.238.190:36281
I20260811 04:38:10.598558 15629 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:10.599190 15629 heartbeater.cc:511] Master 127.2.238.190:36281 requested a full tablet report, sending...
I20260811 04:38:10.600389 15111 ts_manager.cc:194] Registered new tserver with Master: 0eecec8c32644f80aac914c2ca5163ee (127.2.238.131:45007)
I20260811 04:38:10.601132 15111 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:55243
I20260811 04:38:10.608824 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:38:10.618741 15348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.611138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.2.238.190:36281)
0811 04:38:10.611266 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.611272 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.611582 (+ 310us) server_negotiation.cc:407] Connection header received
0811 04:38:10.611707 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.611711 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.611762 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.611846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:10.612561 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.613039 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.613845 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.614047 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.614156 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.615610 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:10.615614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:10.615616 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:10.615639 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:10.617029 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.617461 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.617469 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.617470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.617544 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:10.617841 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:10.617845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:10.617846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:10.618056 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:10.618194 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.618421 (+ 227us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.618535 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 04:38:10.621766 15111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:38:10.625991 15111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:38:10.645714 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.635965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45007 (local address 127.0.0.1:50504)
0811 04:38:10.638555 (+ 2590us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.638602 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.638729 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.639460 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.639466 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.639491 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:10.639619 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.639625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.641134 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.641138 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.642899 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.642907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.645181 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.645184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.645290 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.645330 (+ 40us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.645434 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.645542 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2418,"thread_start_us":103,"threads_started":1}
I20260811 04:38:10.646741 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.636106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37543 (local address 127.0.0.1:34004)
0811 04:38:10.639646 (+ 3540us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.639658 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.639748 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.642230 (+ 2482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.642233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.642251 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:10.642351 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.642357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.644090 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.644093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.645020 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.645030 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.646371 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.646374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.646464 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.646483 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.646555 (+ 72us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.646628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3482,"thread_start_us":82,"threads_started":1}
I20260811 04:38:10.646919 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.636939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.2.238.131:45007)
0811 04:38:10.638996 (+ 2057us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.639009 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.639031 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:38:10.639098 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.639103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.639297 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.639437 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:10.639754 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.640978 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.643040 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.643791 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.643890 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.644024 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.646095 (+ 2071us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.646201 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1956,"thread_start_us":94,"threads_started":1}
I20260811 04:38:10.647133 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.637278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.2.238.129:37543)
0811 04:38:10.637755 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.637763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.641872 (+ 4109us) server_negotiation.cc:407] Connection header received
0811 04:38:10.641956 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.641960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.642086 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.642245 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:10.643059 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.643998 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.645149 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.646259 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.646360 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.646489 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.646586 (+ 97us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.646673 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":183,"threads_started":1}
I20260811 04:38:10.650084 15272 tablet_service.cc:1467] Processing CreateTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 (DEFAULT_TABLE table=test-table [id=f92e3df55c6f4f29b50868c81450a20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:10.650087 15561 tablet_service.cc:1467] Processing CreateTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 (DEFAULT_TABLE table=test-table [id=f92e3df55c6f4f29b50868c81450a20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:10.651285 15561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956eda1e39704e1f8b5cf91d98388b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:10.651285 15272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956eda1e39704e1f8b5cf91d98388b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:10.653762 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.637063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:37077 (local address 127.0.0.1:38630)
0811 04:38:10.639927 (+ 2864us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.639938 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.639999 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.641404 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.641407 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.641425 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:10.641518 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.641523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.651504 (+ 9981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.651507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.652499 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.652509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.653439 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.653443 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.653519 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.653540 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.653580 (+ 40us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.653635 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2814,"thread_start_us":105,"threads_started":1}
I20260811 04:38:10.654443 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.637817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38630 (local address 127.2.238.130:37077)
0811 04:38:10.640213 (+ 2396us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.640228 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.640258 (+ 30us) server_negotiation.cc:407] Connection header received
0811 04:38:10.640329 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.640334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.641251 (+ 917us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.641673 (+ 422us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:10.650311 (+ 8638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.651371 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.652660 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.653320 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.654029 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.654171 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.654232 (+ 61us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.654322 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2295,"thread_start_us":171,"threads_started":1}
I20260811 04:38:10.657490 15419 tablet_service.cc:1467] Processing CreateTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 (DEFAULT_TABLE table=test-table [id=f92e3df55c6f4f29b50868c81450a20f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:10.658608 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 956eda1e39704e1f8b5cf91d98388b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:10.662170 15647 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Bootstrap starting.
I20260811 04:38:10.664114 15647 tablet_bootstrap.cc:654] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:10.664564 15648 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Bootstrap starting.
I20260811 04:38:10.665021 15647 log.cc:824] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:10.667239 15650 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Bootstrap starting.
I20260811 04:38:10.667405 15647 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: No bootstrap required, opened a new log
I20260811 04:38:10.667542 15648 tablet_bootstrap.cc:654] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:10.667644 15647 ts_tablet_manager.cc:1397] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:38:10.668263 15648 log.cc:824] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:10.669502 15650 tablet_bootstrap.cc:654] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:10.670331 15650 log.cc:824] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:10.671491 15648 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: No bootstrap required, opened a new log
I20260811 04:38:10.671669 15648 ts_tablet_manager.cc:1397] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260811 04:38:10.672151 15650 tablet_bootstrap.cc:492] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: No bootstrap required, opened a new log
I20260811 04:38:10.672017 15647 raft_consensus.cc:348] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.672276 15647 raft_consensus.cc:374] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:10.672329 15650 ts_tablet_manager.cc:1397] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260811 04:38:10.672329 15647 raft_consensus.cc:729] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f54c89c7e84643559069d0918dae7050, State: Initialized, Role: FOLLOWER
I20260811 04:38:10.672786 15647 consensus_queue.cc:260] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.673702 15647 ts_tablet_manager.cc:1428] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:38:10.674069 15344 heartbeater.cc:503] Master 127.2.238.190:36281 was elected leader, sending a full tablet report...
I20260811 04:38:10.675798 15648 raft_consensus.cc:348] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.676039 15648 raft_consensus.cc:374] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:10.676105 15648 raft_consensus.cc:729] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eecec8c32644f80aac914c2ca5163ee, State: Initialized, Role: FOLLOWER
I20260811 04:38:10.676147 15650 raft_consensus.cc:348] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.676403 15650 raft_consensus.cc:374] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:10.676452 15650 raft_consensus.cc:729] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180bcb75123946f88e41861e388cbca0, State: Initialized, Role: FOLLOWER
I20260811 04:38:10.676535 15648 consensus_queue.cc:260] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.676882 15650 consensus_queue.cc:260] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.677636 15648 ts_tablet_manager.cc:1428] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:38:10.677851 15650 ts_tablet_manager.cc:1428] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Time spent starting tablet: real 0.005s user 0.005s sys 0.002s
I20260811 04:38:10.678170 15629 heartbeater.cc:503] Master 127.2.238.190:36281 was elected leader, sending a full tablet report...
I20260811 04:38:10.678206 15489 heartbeater.cc:503] Master 127.2.238.190:36281 was elected leader, sending a full tablet report...
W20260811 04:38:10.683501 15490 tablet.cc:2369] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 11 04:38:10 dist-test-slave-0kls krb5kdc[14611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423089, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
W20260811 04:38:10.828406 15630 tablet.cc:2369] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:38:10.841986 15345 tablet.cc:2369] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:38:10.875355 15654 raft_consensus.cc:482] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:38:10.875641 15654 raft_consensus.cc:504] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.877169 15654 leader_election.cc:288] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f54c89c7e84643559069d0918dae7050 (127.2.238.129:37543), 180bcb75123946f88e41861e388cbca0 (127.2.238.130:37077)
I20260811 04:38:10.882743 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.877349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37543 (local address 127.2.238.131:60207)
0811 04:38:10.877700 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.877716 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.877889 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.878769 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.878772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.878793 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:10.878916 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.878923 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.879956 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.879962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.881236 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.881247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.882299 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.882303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.882416 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.882441 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.882517 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.882599 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248}
I20260811 04:38:10.883455 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.878118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:60207 (local address 127.2.238.129:37543)
0811 04:38:10.878246 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.878250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.878266 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:38:10.878323 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.878331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.878400 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.878470 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:10.879017 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.879815 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.881381 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.882034 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.882180 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.882239 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.883228 (+ 989us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.883317 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 04:38:10.884057 15293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956eda1e39704e1f8b5cf91d98388b86" candidate_uuid: "0eecec8c32644f80aac914c2ca5163ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54c89c7e84643559069d0918dae7050" is_pre_election: true
I20260811 04:38:10.884411 15293 raft_consensus.cc:2454] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eecec8c32644f80aac914c2ca5163ee in term 0.
I20260811 04:38:10.884972 15514 leader_election.cc:302] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eecec8c32644f80aac914c2ca5163ee, f54c89c7e84643559069d0918dae7050; no voters:
I20260811 04:38:10.885290 15654 raft_consensus.cc:2781] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:38:10.885401 15654 raft_consensus.cc:482] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:38:10.885454 15654 raft_consensus.cc:3037] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:10.886936 15654 raft_consensus.cc:504] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.887178 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.877949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:56133 (local address 127.2.238.130:37077)
0811 04:38:10.878088 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.878094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.882992 (+ 4898us) server_negotiation.cc:407] Connection header received
0811 04:38:10.883111 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.883114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.883174 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.883268 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:10.883659 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.884370 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.885768 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.886401 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.886528 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.886581 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.886980 (+ 399us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.887066 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 04:38:10.887291 15654 leader_election.cc:288] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [CANDIDATE]: Term 1 election: Requested vote from peers f54c89c7e84643559069d0918dae7050 (127.2.238.129:37543), 180bcb75123946f88e41861e388cbca0 (127.2.238.130:37077)
I20260811 04:38:10.887457 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.877665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:37077 (local address 127.2.238.131:56133)
0811 04:38:10.882913 (+ 5248us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:10.882926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:10.883004 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:10.883388 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:10.883392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:10.883411 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:10.883495 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.883502 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.884537 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.884541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.885604 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:10.885617 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.886641 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.886644 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:10.886728 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.886751 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:38:10.886821 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:38:10.886880 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5192,"thread_start_us":281,"threads_started":1}
I20260811 04:38:10.888036 15293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956eda1e39704e1f8b5cf91d98388b86" candidate_uuid: "0eecec8c32644f80aac914c2ca5163ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f54c89c7e84643559069d0918dae7050"
I20260811 04:38:10.888175 15293 raft_consensus.cc:3037] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:10.888473 15436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956eda1e39704e1f8b5cf91d98388b86" candidate_uuid: "0eecec8c32644f80aac914c2ca5163ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180bcb75123946f88e41861e388cbca0" is_pre_election: true
I20260811 04:38:10.888767 15436 raft_consensus.cc:2454] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0eecec8c32644f80aac914c2ca5163ee in term 0.
I20260811 04:38:10.889125 15437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "956eda1e39704e1f8b5cf91d98388b86" candidate_uuid: "0eecec8c32644f80aac914c2ca5163ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "180bcb75123946f88e41861e388cbca0"
I20260811 04:38:10.889240 15437 raft_consensus.cc:3037] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:10.889530 15293 raft_consensus.cc:2454] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eecec8c32644f80aac914c2ca5163ee in term 1.
I20260811 04:38:10.889992 15514 leader_election.cc:302] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eecec8c32644f80aac914c2ca5163ee, f54c89c7e84643559069d0918dae7050; no voters:
I20260811 04:38:10.890244 15654 raft_consensus.cc:2781] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:38:10.890504 15654 raft_consensus.cc:686] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 1 LEADER]: Becoming Leader. State: Replica: 0eecec8c32644f80aac914c2ca5163ee, State: Running, Role: LEADER
I20260811 04:38:10.890635 15437 raft_consensus.cc:2454] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0eecec8c32644f80aac914c2ca5163ee in term 1.
I20260811 04:38:10.890946 15654 consensus_queue.cc:237] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } }
I20260811 04:38:10.894377 15109 catalog_manager.cc:5495] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 0eecec8c32644f80aac914c2ca5163ee (127.2.238.131). New cstate: current_term: 1 leader_uuid: "0eecec8c32644f80aac914c2ca5163ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eecec8c32644f80aac914c2ca5163ee" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45007 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 } health_report { overall_health: UNKNOWN } } }
I20260811 04:38:10.950512 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:10.945619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.2.238.131:45007)
0811 04:38:10.945757 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:10.945761 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:10.945775 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:38:10.945884 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:10.945888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:10.945964 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:10.946079 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:38:10.946399 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.947230 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.948427 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:10.948756 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:10.948846 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:10.949240 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:38:10.949757 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:38:10.949902 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:10.950295 (+ 393us) server_negotiation.cc:299] Negotiation successful
0811 04:38:10.950354 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 04:38:10.957271 15437 raft_consensus.cc:1261] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 1 FOLLOWER]: Refusing update from remote peer 0eecec8c32644f80aac914c2ca5163ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:38:10.958170 15654 consensus_queue.cc:1035] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "180bcb75123946f88e41861e388cbca0" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 37077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:38:10.965101 15293 raft_consensus.cc:1261] T 956eda1e39704e1f8b5cf91d98388b86 P f54c89c7e84643559069d0918dae7050 [term 1 FOLLOWER]: Refusing update from remote peer 0eecec8c32644f80aac914c2ca5163ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:38:10.965713 15654 consensus_queue.cc:1035] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f54c89c7e84643559069d0918dae7050" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:38:10.976712 15662 mvcc.cc:205] Tried to move back new op lower bound from 7317188980553125888 to 7317188980292538368. Current Snapshot: MvccSnapshot[applied={T|T < 7317188980553125888}]
I20260811 04:38:10.989615 15109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:38:10.989827 15109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:38:10.992856 15109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P db973d359e87484dbe1a66a953586ba7: Sending DeleteTablet for 3 replicas of tablet 956eda1e39704e1f8b5cf91d98388b86
I20260811 04:38:10.993894 15419 tablet_service.cc:1514] Processing DeleteTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:10 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:38630
I20260811 04:38:10.994539 15272 tablet_service.cc:1514] Processing DeleteTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:10 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:34004
I20260811 04:38:10.995492 15561 tablet_service.cc:1514] Processing DeleteTablet for tablet 956eda1e39704e1f8b5cf91d98388b86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:10 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:50504
I20260811 04:38:10.996865 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 15202
I20260811 04:38:11.004644 15674 tablet_replica.cc:330] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: stopping tablet replica
I20260811 04:38:11.005117 15674 raft_consensus.cc:2229] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:38:11.005510 15671 tablet_replica.cc:330] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: stopping tablet replica
I20260811 04:38:11.007249 15671 raft_consensus.cc:2229] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:38:11.007658 15674 raft_consensus.cc:2258] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:38:11.014349 15671 raft_consensus.cc:2258] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 04:38:11.015120 15098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:37543 (error 108)
I20260811 04:38:11.016327 15674 ts_tablet_manager.cc:1905] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:38:11.016805 15671 ts_tablet_manager.cc:1905] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:38:11.019626 15674 ts_tablet_manager.cc:1918] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:38:11.019821 15674 log.cc:1196] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/956eda1e39704e1f8b5cf91d98388b86
I20260811 04:38:11.019927 15671 ts_tablet_manager.cc:1918] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:38:11.020131 15671 log.cc:1196] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/956eda1e39704e1f8b5cf91d98388b86
I20260811 04:38:11.020237 15674 ts_tablet_manager.cc:1939] T 956eda1e39704e1f8b5cf91d98388b86 P 0eecec8c32644f80aac914c2ca5163ee: Deleting consensus metadata
I20260811 04:38:11.020565 15671 ts_tablet_manager.cc:1939] T 956eda1e39704e1f8b5cf91d98388b86 P 180bcb75123946f88e41861e388cbca0: Deleting consensus metadata
I20260811 04:38:11.021621 15097 catalog_manager.cc:4841] TS 0eecec8c32644f80aac914c2ca5163ee (127.2.238.131:45007): tablet 956eda1e39704e1f8b5cf91d98388b86 (table test-table [id=f92e3df55c6f4f29b50868c81450a20f]) successfully deleted
I20260811 04:38:11.021871 15098 catalog_manager.cc:4841] TS 180bcb75123946f88e41861e388cbca0 (127.2.238.130:37077): tablet 956eda1e39704e1f8b5cf91d98388b86 (table test-table [id=f92e3df55c6f4f29b50868c81450a20f]) successfully deleted
I20260811 04:38:11.023217 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:11.022940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:37543 (local address 127.0.0.1:34008)
0811 04:38:11.023052 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:11.023126 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:37543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260811 04:38:11.023208 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 15353
W20260811 04:38:11.023428 15098 catalog_manager.cc:4584] TS f54c89c7e84643559069d0918dae7050 (127.2.238.129:37543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 956eda1e39704e1f8b5cf91d98388b86: Network error: Client connection negotiation failed: client connection to 127.2.238.129:37543: connect: Connection refused (error 111)
I20260811 04:38:11.033703 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 15494
I20260811 04:38:11.042012 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 15079
I20260811 04:38:11.050175 3002 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 04:38:11.394584 3002 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 04:38:11.394717 3002 mini_ranger.cc:66] Stopping Ranger...
I20260811 04:38:11.740435 3002 mini_ranger.cc:68] Stopped Ranger
2026-08-11 04:38:11.740 UTC [14643] LOG: received smart shutdown request
2026-08-11 04:38:11.744 UTC [14643] LOG: background worker "logical replication launcher" (PID 14652) exited with exit code 1
2026-08-11 04:38:11.744 UTC [14647] LOG: shutting down
2026-08-11 04:38:11.896 UTC [14643] LOG: database system is shut down
2026-08-11T04:38:11Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45511 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:38:12 dist-test-slave-0kls krb5kdc[15685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:38:12 dist-test-slave-0kls krb5kdc[15685](info): set up 2 sockets
Aug 11 04:38:12 dist-test-slave-0kls krb5kdc[15685](info): commencing operation
krb5kdc: starting...
W20260811 04:38:13.055136 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:38:13 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:38:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:38:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangeradmin_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangeradmin_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangerlookup_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangerlookup_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
I20260811 04:38:13.220141 3002 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taski3hQIv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/postgres -l logfile start
2026-08-11 04:38:15.574 UTC [15718] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-11 04:38:15.574 UTC [15718] LOG: listening on IPv4 address "127.2.238.148", port 56277
2026-08-11 04:38:15.578 UTC [15718] LOG: listening on Unix socket "/tmp/.s.PGSQL.56277"
2026-08-11 04:38:15.613 UTC [15721] LOG: database system was shut down at 2026-08-11 04:38:14 UTC
2026-08-11 04:38:15.618 UTC [15718] LOG: database system is ready to accept connections
I20260811 04:38:16.585203 3002 mini_postgres.cc:96] Postgres bound to 56277
2026-08-11 04:38:16.592 UTC [15729] FATAL: database "slave" does not exist
127.2.238.148:56277 - accepting connections
I20260811 04:38:16.593156 3002 mini_ranger.cc:161] Starting Ranger...
I20260811 04:38:16.609731 3002 mini_ranger.cc:84] Created miniranger Postgres user
I20260811 04:38:17.092653 3002 mini_ranger.cc:87] Created ranger Postgres database
I20260811 04:38:17.092741 3002 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin
2026-08-11 04:38:17,375 [I] DB FLAVOR :POSTGRES
2026-08-11 04:38:17,375 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 04:38:17,375 [I] Checking connection..
2026-08-11 04:38:17,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-11 04:38:17,623 [I] Checking connection passed.
2026-08-11 04:38:17,623 [I] --------- Verifying version history table ---------
2026-08-11 04:38:17,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:38:17,885 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 04:38:17,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:38:18,124 [I] Table x_db_version_h does not exist in database ranger
2026-08-11 04:38:18,124 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-11 04:38:18,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-11 04:38:18.348 UTC [15797] WARNING: there is no transaction in progress
2026-08-11 04:38:18,355 [I] create_dbversion_catalog.sql file imported successfully
2026-08-11 04:38:18,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-11 04:38:18,614 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-11 04:38:18,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 04:38:18,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-11 04:38:18,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-11 04:38:19,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-0kls.c.gcp-upstream.internal','N') ;"
2026-08-11 04:38:19,589 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 04:38:19,856 [I] Table x_portal_user does not exist in database ranger
2026-08-11 04:38:19,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 04:38:20,116 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-11 04:38:20,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 04:38:20,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 04:38:20,623 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-11 04:38:20,623 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-11 04:38:21.563 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.578 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.594 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.609 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.632 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.648 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.701 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.707 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.721 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.731 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.741 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.760 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.769 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.775 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.781 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:21.788 UTC [15916] WARNING: there is no transaction in progress
2026-08-11 04:38:22,404 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-11 04:38:22,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-11 04:38:22,658 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-11 04:38:22,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-11 04:38:22,917 [I] Table x_portal_user already exists in database 'ranger'
2026-08-11 04:38:22,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-11 04:38:23,144 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-11 04:38:23,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-11 04:38:23,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-11 04:38:23,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-0kls.c.gcp-upstream.internal';"
2026-08-11 04:38:23,866 [I] CORE_DB_SCHEMA import status has been updated
2026-08-11 04:38:23,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-11 04:38:24,108 [I] DB_PATCHES have already been applied
I20260811 04:38:24.114069 3002 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taski3hQIv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260811 04:38:24.114149 3002 mini_ranger.cc:193] Using host: 127.2.238.148
I20260811 04:38:24.118404 3002 mini_ranger.cc:227] Ranger admin URL: http://127.2.238.148:35751
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-admin/ews/webapp].
Aug 11, 2026 4:38:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260811 04:38:25.236519 3002 mini_ranger.cc:160] Time spent starting Ranger: real 8.643s user 0.000s sys 0.011s
Aug 11, 2026 4:38:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 11, 2026 4:38:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 11, 2026 4:38:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 11, 2026 4:38:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 11, 2026 4:38:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260811 04:38:45.129139 3002 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.238.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.238.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangerkms_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.238.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/rangerkms_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.238.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.238.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.238.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/HTTP_127.2.238.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 11 04:38:45 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423125, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260811 04:38:45.225332 3002 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260811 04:38:45.236358 3002 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260811 04:38:46.096573 3002 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260811 04:38:46.096652 3002 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms
I20260811 04:38:46.096675 3002 mini_ranger_kms.cc:227] Using postgres at 127.2.238.148:56277
2026-08-11 04:38:46,265 [I] DB FLAVOR :POSTGRES
2026-08-11 04:38:46,265 [I] --------- Verifying Ranger DB connection ---------
2026-08-11 04:38:46,265 [I] Checking connection
2026-08-11 04:38:46,265 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-11 04:38:46,534 [I] connection success
2026-08-11 04:38:46,534 [I] --------- Verifying Ranger DB tables ---------
2026-08-11 04:38:46,534 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-11 04:38:46,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-11 04:38:46,818 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-11 04:38:46,819 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-11 04:38:46,819 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-11 04:38:46,819 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.238.148:56277/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-11 04:38:47,095 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260811 04:38:47.573961 3002 mini_ranger_kms.cc:310] Created kms service
I20260811 04:38:47.769395 3002 mini_ranger_kms.cc:326] Created kudu user
I20260811 04:38:47.820881 3002 mini_ranger_kms.cc:343] Created rangerkms user
Aug 11 04:38:47 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423125, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:48.235497 3002 mini_ranger_kms.cc:384] Added ranger policy
I20260811 04:38:48.235740 3002 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms:/tmp/dist-test-taski3hQIv/build/debug/bin/postgresql.jar:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taski3hQIv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260811 04:38:48.235766 3002 mini_ranger_kms.cc:240] Using host: 127.2.238.148
I20260811 04:38:48.240549 3002 mini_ranger_kms.cc:276] Ranger KMS PID: 16108
I20260811 04:38:48.240643 3002 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.238.148:54895
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-kms/ews/webapp].
Aug 11, 2026 4:38:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 11 04:38:50 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786423130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:50.554450 3002 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.329s user 0.000s sys 0.154s
I20260811 04:38:50.554595 3002 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260811 04:38:50.554692 3002 mini_ranger_kms.cc:401] 127.2.238.148:54895/kms/v1/keys
Aug 11 04:38:50 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
Aug 11 04:38:51 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423130, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.238.148@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
Aug 11 04:38:53 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:38:53.546100 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46163
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46163
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:53.653921 16163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:53.654173 16163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:53.654209 16163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:53.657958 16163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:38:53.658010 16163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:53.658028 16163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:53.658046 16163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:38:53.658061 16163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:38:53.663393 16163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.238.148:54895/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46163
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:53.664362 16163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:53.665398 16163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:38:53.674073 16163 server_base.cc:1034] running on GCE node
W20260811 04:38:53.674201 16171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:53.674171 16168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:53.674499 16169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:53.675031 16163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:53.676086 16163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:53.677295 16163 hybrid_clock.cc:648] HybridClock initialized: now 1786423133677274 us; error 30 us; skew 500 ppm
Aug 11 04:38:53 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423133, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:53.683703 16163 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:38:53.684623 16163 webserver.cc:460] Webserver started at http://127.2.238.190:35999/ using document root <none> and password file <none>
I20260811 04:38:53.685207 16163 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:53.685293 16163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:53.685539 16163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:53 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423133, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:53.747169 16163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "11595274755d472eba945c3e6acb3505"
format_stamp: "Formatted at 2026-08-11 04:38:53 on dist-test-slave-0kls"
server_key: "a66e7e20bd60b26d6b8c0faa95439735"
server_key_iv: "1db22a0a2fe65f22bb0035c642daec9d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:53.747843 16163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "11595274755d472eba945c3e6acb3505"
format_stamp: "Formatted at 2026-08-11 04:38:53 on dist-test-slave-0kls"
server_key: "a66e7e20bd60b26d6b8c0faa95439735"
server_key_iv: "1db22a0a2fe65f22bb0035c642daec9d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:53.752223 16163 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:38:53.770624 16178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:53.772044 16163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:38:53.772199 16163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "11595274755d472eba945c3e6acb3505"
format_stamp: "Formatted at 2026-08-11 04:38:53 on dist-test-slave-0kls"
server_key: "a66e7e20bd60b26d6b8c0faa95439735"
server_key_iv: "1db22a0a2fe65f22bb0035c642daec9d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:53.772338 16163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:53.785267 16163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:53.788273 16163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:53.788473 16163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:53.797299 16163 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:46163
I20260811 04:38:53.797346 16232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:46163 every 8 connection(s)
I20260811 04:38:53.798399 16163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:38:53.801908 16233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:53.803125 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16163
I20260811 04:38:53.803260 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:38:53.803550 3002 external_mini_cluster.cc:1428] Setting key 8c44540a974a984741a62580bf69bd1f
I20260811 04:38:53.808844 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Bootstrap starting.
I20260811 04:38:53.811641 16233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Neither blocks nor log segments found. Creating new log.
Aug 11 04:38:53 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:53.812489 16233 log.cc:824] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:53.814966 16233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: No bootstrap required, opened a new log
I20260811 04:38:53.818533 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:53.804995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.2.238.190:46163)
0811 04:38:53.805381 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:53.805398 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:53.805444 (+ 46us) server_negotiation.cc:407] Connection header received
0811 04:38:53.806109 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:53.806127 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:53.806474 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:53.806788 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:53.807543 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:53.808543 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:53.809453 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:53.809957 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:53.810121 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:53.812640 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:53.812645 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:53.812658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:53.812682 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:53.815228 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:53.815733 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:53.815739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:53.815744 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:53.815866 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:53.816218 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:53.816221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:53.816223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:53.816713 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:53.816913 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:53.817228 (+ 315us) server_negotiation.cc:299] Negotiation successful
0811 04:38:53.817519 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":126,"threads_started":1}
I20260811 04:38:53.818871 16233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:53.819144 16233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:53.819247 16233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11595274755d472eba945c3e6acb3505, State: Initialized, Role: FOLLOWER
I20260811 04:38:53.819701 16233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:53.819904 16233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:38:53.819994 16233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:38:53.820094 16233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:53.821328 16233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:53.821674 16233 leader_election.cc:302] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11595274755d472eba945c3e6acb3505; no voters:
I20260811 04:38:53.822018 16233 leader_election.cc:288] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:38:53.822149 16240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:38:53.822369 16240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 LEADER]: Becoming Leader. State: Replica: 11595274755d472eba945c3e6acb3505, State: Running, Role: LEADER
I20260811 04:38:53.822714 16240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:53.823228 16233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:38:53.823843 16240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11595274755d472eba945c3e6acb3505. Latest consensus state: current_term: 1 leader_uuid: "11595274755d472eba945c3e6acb3505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } } }
I20260811 04:38:53.823951 16240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:53.824285 16246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:38:53.824499 16242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11595274755d472eba945c3e6acb3505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } } }
I20260811 04:38:53.824775 16242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:53.827452 16246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:38:53.827890 16233 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 04:38:53.838742 16246 catalog_manager.cc:1347] Generated new cluster ID: fbf5cefc1d1d4d9b8ae4e8c88114af13
I20260811 04:38:53.838835 16246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:38:53.844585 16246 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:38:53.845518 16246 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:38:53.855161 16246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Generated new TSK 0
I20260811 04:38:53.855845 16246 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:38:54.556466 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:54.670511 16281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:54.670743 16281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:54.670809 16281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:54.675177 16281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:54.675259 16281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:54.675374 16281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:38:54.681327 16281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:54.682425 16281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:54.683674 16281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:54.695724 16290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:54.695875 16293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:54.695717 16289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:54.696938 16281 server_base.cc:1034] running on GCE node
I20260811 04:38:54.697316 16281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:54.698000 16281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:54.699191 16281 hybrid_clock.cc:648] HybridClock initialized: now 1786423134699167 us; error 45 us; skew 500 ppm
Aug 11 04:38:54 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423134, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:54.702901 16281 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:38:54.703986 16281 webserver.cc:460] Webserver started at http://127.2.238.129:41155/ using document root <none> and password file <none>
I20260811 04:38:54.704665 16281 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:54.704770 16281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:54.705070 16281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:54 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423134, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:54.721637 16281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "a1235cd6d6934ddba2a38a5ea07ebc31"
format_stamp: "Formatted at 2026-08-11 04:38:54 on dist-test-slave-0kls"
server_key: "b158a3d9e45bf40f34bb9425307ba5aa"
server_key_iv: "845033bc95cb9b242a88241b8b9a1b04"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:54.722437 16281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "a1235cd6d6934ddba2a38a5ea07ebc31"
format_stamp: "Formatted at 2026-08-11 04:38:54 on dist-test-slave-0kls"
server_key: "b158a3d9e45bf40f34bb9425307ba5aa"
server_key_iv: "845033bc95cb9b242a88241b8b9a1b04"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:54.731226 16281 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260811 04:38:54.748221 16300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:54.749933 16281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:38:54.750077 16281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "a1235cd6d6934ddba2a38a5ea07ebc31"
format_stamp: "Formatted at 2026-08-11 04:38:54 on dist-test-slave-0kls"
server_key: "b158a3d9e45bf40f34bb9425307ba5aa"
server_key_iv: "845033bc95cb9b242a88241b8b9a1b04"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:54.750208 16281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:54.770262 16281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:54.777350 16281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:54.777592 16281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:54.778365 16281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:54.779433 16281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:54.779526 16281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:54.779606 16281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:54.779652 16281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:54.790318 16281 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40511
I20260811 04:38:54.790596 16422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40511 every 8 connection(s)
I20260811 04:38:54.791443 16281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:38:54.793340 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16281
I20260811 04:38:54.793454 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:38:54.793785 3002 external_mini_cluster.cc:1428] Setting key 9b7289f3ce71de251e91be0f1a518f80
Aug 11 04:38:54 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423134, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:54.812412 16426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:54.793978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.129:34055)
0811 04:38:54.794661 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:54.794712 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:54.795657 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:54.799812 (+ 4155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:54.799820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:54.800175 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:54.800824 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:54.800843 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:54.802043 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:54.802047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:54.803368 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:54.803380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:54.804128 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:54.804132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:54.804322 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:54.805019 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:54.805042 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:54.806771 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:54.809024 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:54.809030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:54.809046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:54.809407 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:54.809757 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:54.809760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:54.809762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:54.809957 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:54.810488 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:54.810496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:54.810807 (+ 311us) client_negotiation.cc:769] Sending connection context
0811 04:38:54.811007 (+ 200us) client_negotiation.cc:240] Negotiation successful
0811 04:38:54.811354 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":157,"threads_started":1}
I20260811 04:38:54.813205 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:54.794341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:34055 (local address 127.2.238.190:46163)
0811 04:38:54.799308 (+ 4967us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:54.799317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:54.799330 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:38:54.799366 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:54.799369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:54.799418 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:54.799505 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:54.801058 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:54.801901 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:54.803526 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:54.803842 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:54.803940 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:54.806961 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:54.806966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:54.806969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:54.806996 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:54.808883 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:54.809532 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:54.809535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:54.809537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:54.809638 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:54.810082 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:54.810086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:54.810088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:54.810331 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:54.810430 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:54.812890 (+ 2460us) server_negotiation.cc:299] Negotiation successful
0811 04:38:54.813031 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4888,"thread_start_us":113,"threads_started":1}
I20260811 04:38:54.816947 16423 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:54.817229 16423 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:54.817936 16423 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:54.819886 16195 ts_manager.cc:194] Registered new tserver with Master: a1235cd6d6934ddba2a38a5ea07ebc31 (127.2.238.129:40511)
I20260811 04:38:54.821439 16195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:34055
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:38:54.844905 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:38:54 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786423134, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260811 04:38:54.970439 16430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:54.970713 16430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:54.970783 16430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:54.974877 16430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:54.974958 16430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:54.975061 16430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:38:54.980358 16430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:54.981546 16430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:54.982918 16430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:54.994546 16437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:54.994858 16430 server_base.cc:1034] running on GCE node
W20260811 04:38:54.995093 16440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:54.995196 16438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:54.995622 16430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:54.996294 16430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:54.997467 16430 hybrid_clock.cc:648] HybridClock initialized: now 1786423134997448 us; error 36 us; skew 500 ppm
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:55.003636 16430 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:38:55.004778 16430 webserver.cc:460] Webserver started at http://127.2.238.130:44321/ using document root <none> and password file <none>
I20260811 04:38:55.005509 16430 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:55.005611 16430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:55.005897 16430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:55.020555 16430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "7c499e6e9a8e441a9d48667a32006091"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "bd10d07d4221a6e6b7716f4e949f6e66"
server_key_iv: "a828532b79e75faf4aa3f1fa2f996e6c"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.021340 16430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "7c499e6e9a8e441a9d48667a32006091"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "bd10d07d4221a6e6b7716f4e949f6e66"
server_key_iv: "a828532b79e75faf4aa3f1fa2f996e6c"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.027680 16430 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260811 04:38:55.050244 16449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.051956 16430 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260811 04:38:55.052105 16430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "7c499e6e9a8e441a9d48667a32006091"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "bd10d07d4221a6e6b7716f4e949f6e66"
server_key_iv: "a828532b79e75faf4aa3f1fa2f996e6c"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.052240 16430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:55.074304 16430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:55.078112 16430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:55.078298 16430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:55.078936 16430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:55.079989 16430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:55.080053 16430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.080121 16430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:55.080152 16430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.100693 16430 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:40289
I20260811 04:38:55.102010 16430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:38:55.102864 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16430
I20260811 04:38:55.102985 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:38:55.103387 3002 external_mini_cluster.cc:1428] Setting key 973afa57680b8ccc9d5b4564beb5444c
I20260811 04:38:55.103755 16570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:40289 every 8 connection(s)
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:55.122368 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:55.105962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41969 (local address 127.2.238.190:46163)
0811 04:38:55.106110 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:55.106114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:55.108250 (+ 2136us) server_negotiation.cc:407] Connection header received
0811 04:38:55.109520 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:55.109523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:55.109578 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:55.109635 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:55.111405 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.112235 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.113963 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.114318 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.114414 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:55.117724 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:55.117728 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:55.117730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:55.117760 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:55.119679 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:55.120373 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:55.120377 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:55.120379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:55.120473 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:55.120924 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:55.120928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:55.120930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:55.121193 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:55.121283 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:55.122008 (+ 725us) server_negotiation.cc:299] Negotiation successful
0811 04:38:55.122161 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:38:55.124200 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:55.106541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.130:41969)
0811 04:38:55.108078 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:55.108150 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:55.109259 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:55.110062 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:55.110071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:55.110497 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:55.111216 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:55.111238 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.112408 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.112411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:55.113793 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:55.113824 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.114625 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.114630 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:55.114821 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:55.115555 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:55.115575 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:55.117528 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:55.119844 (+ 2316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:55.119850 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:55.119867 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:55.120238 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:55.120595 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:55.120598 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:55.120600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:55.120797 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:55.121357 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:55.121364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:55.121679 (+ 315us) client_negotiation.cc:769] Sending connection context
0811 04:38:55.122635 (+ 956us) client_negotiation.cc:240] Negotiation successful
0811 04:38:55.122974 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"thread_start_us":164,"threads_started":1}
I20260811 04:38:55.125702 16572 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:55.125989 16572 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:55.126629 16572 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:55.127933 16195 ts_manager.cc:194] Registered new tserver with Master: 7c499e6e9a8e441a9d48667a32006091 (127.2.238.130:40289)
I20260811 04:38:55.128692 16195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:41969
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:38:55.169385 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:55.340512 16579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:55.340895 16579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:55.340956 16579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:55.347007 16579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:55.347087 16579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:55.347168 16579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:38:55.354610 16579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:55.356048 16579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:55.357304 16579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:55.370227 16586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:55.370376 16589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:55.370581 16585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:55.371816 16579 server_base.cc:1034] running on GCE node
I20260811 04:38:55.372464 16579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:55.373226 16579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:55.374375 16579 hybrid_clock.cc:648] HybridClock initialized: now 1786423135374306 us; error 86 us; skew 500 ppm
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:55.378356 16579 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:38:55.379401 16579 webserver.cc:460] Webserver started at http://127.2.238.131:46547/ using document root <none> and password file <none>
I20260811 04:38:55.380134 16579 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:55.380205 16579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:38:55.380437 16579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:55.400645 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "01a444c20d03ec7964b73fbf1464aeca"
server_key_iv: "15a019c26e06f034874f4e8016da2e30"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.401391 16579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "01a444c20d03ec7964b73fbf1464aeca"
server_key_iv: "15a019c26e06f034874f4e8016da2e30"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.407217 16579 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423134, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:55.430258 16599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.431787 16579 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260811 04:38:55.431903 16579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "01a444c20d03ec7964b73fbf1464aeca"
server_key_iv: "15a019c26e06f034874f4e8016da2e30"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.432029 16579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:55.474550 16579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:55.480190 16579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:55.480393 16579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:55.481050 16579 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:55.482139 16579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:55.482203 16579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.482261 16579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:55.482290 16579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.493645 16579 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:44227
I20260811 04:38:55.494791 16579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:38:55.499092 16728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:44227 every 8 connection(s)
I20260811 04:38:55.502135 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16579
I20260811 04:38:55.502277 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:38:55.502622 3002 external_mini_cluster.cc:1428] Setting key 2b8e6ee82729c6534e9d15953e4e84e0
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:55.529825 16427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:55.503379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40057 (local address 127.2.238.190:46163)
0811 04:38:55.504461 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:55.504465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:55.504481 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:38:55.504976 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:55.504979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:55.505036 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:55.505087 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:55.506740 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.507562 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.509245 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.509590 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.509661 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:55.522343 (+ 12682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:55.522348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:55.522350 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:55.522381 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:55.524251 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:55.527889 (+ 3638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:55.527893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:55.527895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:55.528005 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:55.528414 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:55.528417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:55.528419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:55.528661 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:55.528740 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:55.529488 (+ 748us) server_negotiation.cc:299] Negotiation successful
0811 04:38:55.529625 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005}
I20260811 04:38:55.531502 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:55.502169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.131:40057)
0811 04:38:55.503602 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:55.503660 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:55.504726 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:55.505498 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:55.505507 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:55.505907 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:55.506559 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:55.506576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.507726 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.507729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:55.509099 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:55.509110 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:55.509891 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:55.509894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:55.510090 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:55.516627 (+ 6537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:55.517867 (+ 1240us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:55.519658 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:55.524409 (+ 4751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:55.524416 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:55.524436 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:55.527770 (+ 3334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:55.528112 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:55.528115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:55.528117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:55.528308 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:55.528815 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:55.528823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:55.529182 (+ 359us) client_negotiation.cc:769] Sending connection context
0811 04:38:55.530060 (+ 878us) client_negotiation.cc:240] Negotiation successful
0811 04:38:55.530427 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1179,"thread_start_us":171,"threads_started":1}
I20260811 04:38:55.533156 16729 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:55.533402 16729 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:55.533999 16729 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:55.536139 16195 ts_manager.cc:194] Registered new tserver with Master: f48b93a8d29645d0a8e7d4b0d31f99b0 (127.2.238.131:44227)
I20260811 04:38:55.536840 16195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:40057
I20260811 04:38:55.547147 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:38:55.547828 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16281
I20260811 04:38:55.559939 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16430
I20260811 04:38:55.569334 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16579
I20260811 04:38:55.579273 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16163
I20260811 04:38:55.587877 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46163
--webserver_interface=127.2.238.190
--webserver_port=35999
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46163
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:38:55.702148 16732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:55.702468 16732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:55.702549 16732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:55.706314 16732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:38:55.706387 16732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:55.706421 16732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:55.706472 16732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:38:55.706506 16732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:38:55.711880 16732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46163
--ranger_config_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=35999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:55.712942 16732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:55.714213 16732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:55.722774 16738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:55.722774 16741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:55.722899 16739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:55.722985 16732 server_base.cc:1034] running on GCE node
I20260811 04:38:55.723707 16732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:55.724800 16732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:55.725952 16732 hybrid_clock.cc:648] HybridClock initialized: now 1786423135725940 us; error 22 us; skew 500 ppm
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:55.728796 16732 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:38:55.729667 16732 webserver.cc:460] Webserver started at http://127.2.238.190:35999/ using document root <none> and password file <none>
I20260811 04:38:55.730242 16732 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:55.730312 16732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 04:38:55 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423135, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:55.811319 16732 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 04:38:55.814232 16749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:55.815742 16732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:38:55.815882 16732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "11595274755d472eba945c3e6acb3505"
format_stamp: "Formatted at 2026-08-11 04:38:53 on dist-test-slave-0kls"
server_key: "a66e7e20bd60b26d6b8c0faa95439735"
server_key_iv: "1db22a0a2fe65f22bb0035c642daec9d"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:55.816466 16732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:55.829205 16732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:55.831670 16732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:55.831858 16732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:55.839190 16732 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:46163
I20260811 04:38:55.839219 16808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:46163 every 8 connection(s)
I20260811 04:38:55.840304 16732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:38:55.844998 16809 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:38:55.846129 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16732
I20260811 04:38:55.846599 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:40511
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:46163
--webserver_port=41155
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:38:55.846963 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Bootstrap starting.
I20260811 04:38:55.863289 16809 log.cc:824] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:55.870057 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:38:55.870457 16809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Bootstrap complete.
I20260811 04:38:55.874430 16809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:55.874809 16809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11595274755d472eba945c3e6acb3505, State: Initialized, Role: FOLLOWER
I20260811 04:38:55.875273 16809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:55.875415 16809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:38:55.875499 16809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:38:55.875613 16809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:38:55.877372 16809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:55.877703 16809 leader_election.cc:302] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11595274755d472eba945c3e6acb3505; no voters:
I20260811 04:38:55.878084 16809 leader_election.cc:288] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 04:38:55.878172 16813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 04:38:55.878361 16813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [term 2 LEADER]: Becoming Leader. State: Replica: 11595274755d472eba945c3e6acb3505, State: Running, Role: LEADER
I20260811 04:38:55.878645 16813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } }
I20260811 04:38:55.879331 16809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:38:55.879582 16815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11595274755d472eba945c3e6acb3505. Latest consensus state: current_term: 2 leader_uuid: "11595274755d472eba945c3e6acb3505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } } }
I20260811 04:38:55.879699 16815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:55.879992 16814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "11595274755d472eba945c3e6acb3505" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11595274755d472eba945c3e6acb3505" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46163 } } }
I20260811 04:38:55.880093 16817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:38:55.880110 16814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505 [sys.catalog]: This master's current role is: LEADER
I20260811 04:38:55.883211 16817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:38:55.883714 16809 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260811 04:38:55.883762 16817 catalog_manager.cc:1259] Loaded cluster ID: fbf5cefc1d1d4d9b8ae4e8c88114af13
I20260811 04:38:55.883818 16817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:38:55.885946 16817 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:38:55.889528 16817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Loaded TSK: 0
I20260811 04:38:55.891283 16817 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:38:55.980201 16811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:55.980477 16811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:55.980559 16811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:55.984834 16811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:55.984911 16811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:55.984994 16811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:38:55.989933 16811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:40511
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=41155
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:55.990860 16811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:55.991760 16811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:56.003549 16847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:56.004562 16848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:56.006208 16850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:56.007849 16811 server_base.cc:1034] running on GCE node
I20260811 04:38:56.008790 16811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:56.010286 16811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:56.013592 16811 hybrid_clock.cc:648] HybridClock initialized: now 1786423136013491 us; error 120 us; skew 500 ppm
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:56.017171 16811 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:38:56.018281 16811 webserver.cc:460] Webserver started at http://127.2.238.129:41155/ using document root <none> and password file <none>
I20260811 04:38:56.018872 16811 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:56.018980 16811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:56.042644 16811 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.003s sys 0.000s
I20260811 04:38:56.046021 16859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.047598 16811 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260811 04:38:56.047744 16811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "a1235cd6d6934ddba2a38a5ea07ebc31"
format_stamp: "Formatted at 2026-08-11 04:38:54 on dist-test-slave-0kls"
server_key: "b158a3d9e45bf40f34bb9425307ba5aa"
server_key_iv: "845033bc95cb9b242a88241b8b9a1b04"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:56.048548 16811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:56.081157 16811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:56.084959 16811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:56.085175 16811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:56.085896 16811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:56.086984 16811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:56.087050 16811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.087114 16811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:56.087139 16811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.097678 16811 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40511
I20260811 04:38:56.097723 16984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40511 every 8 connection(s)
I20260811 04:38:56.098690 16811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:38:56.106050 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16811
I20260811 04:38:56.106498 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:40289
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:46163
--webserver_port=44321
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:56.117069 16988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.101154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.129:40623)
0811 04:38:56.101645 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:56.101685 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:56.102513 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:56.103676 (+ 1163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:56.103684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:56.104019 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:56.104472 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.104486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.105935 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.105939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.107395 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.107408 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.108602 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.108609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.108835 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.109534 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:56.109556 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:56.111201 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:56.113482 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.113487 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.113501 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.113775 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.114229 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.114232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.114234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.114429 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.115211 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:56.115218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:56.115521 (+ 303us) client_negotiation.cc:769] Sending connection context
0811 04:38:56.115732 (+ 211us) client_negotiation.cc:240] Negotiation successful
0811 04:38:56.116066 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":153,"threads_started":1}
I20260811 04:38:56.119289 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.100947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:40623 (local address 127.2.238.190:46163)
0811 04:38:56.101398 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:56.101425 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:56.101831 (+ 406us) server_negotiation.cc:407] Connection header received
0811 04:38:56.102976 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:56.102987 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:56.103271 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:56.103605 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:56.104648 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.105690 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.107543 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.108384 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.109272 (+ 888us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.111411 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:56.111416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:56.111429 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:56.111455 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:56.113355 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.113951 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.113958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.113964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.114108 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.114548 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.114551 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.114553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.115020 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:56.115151 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:56.118079 (+ 2928us) server_negotiation.cc:299] Negotiation successful
0811 04:38:56.118389 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
I20260811 04:38:56.122107 16985 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:56.122349 16985 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.122861 16985 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
W20260811 04:38:56.124927 16985 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 04:38:56.125059 16985 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.125221 16985 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:56.126025 16985 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.126206 16985 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
W20260811 04:38:56.126904 16985 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:38:56.286417 16989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:56.286659 16989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:56.286700 16989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:56.291275 16989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:56.291329 16989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:56.291409 16989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:38:56.297329 16989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:40289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=44321
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:56.298657 16989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:56.300040 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:56.310357 16995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:56.310384 16994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:56.310544 16989 server_base.cc:1034] running on GCE node
W20260811 04:38:56.310477 16997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:56.311113 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:56.311764 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:56.312983 16989 hybrid_clock.cc:648] HybridClock initialized: now 1786423136312953 us; error 40 us; skew 500 ppm
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:56.316318 16989 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:38:56.317348 16989 webserver.cc:460] Webserver started at http://127.2.238.130:44321/ using document root <none> and password file <none>
I20260811 04:38:56.318109 16989 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:56.318210 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:56.344902 16989 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.000s
I20260811 04:38:56.347473 17005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.349000 16989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260811 04:38:56.349153 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "7c499e6e9a8e441a9d48667a32006091"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "bd10d07d4221a6e6b7716f4e949f6e66"
server_key_iv: "a828532b79e75faf4aa3f1fa2f996e6c"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:56.349781 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:56.368377 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:56.371803 16989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:56.371999 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:56.372674 16989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:56.373708 16989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:56.373811 16989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.373876 16989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:56.373919 16989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.385102 16989 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:40289
I20260811 04:38:56.385249 17130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:40289 every 8 connection(s)
I20260811 04:38:56.387153 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:38:56.393628 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 16989
I20260811 04:38:56.394176 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.238.148:54895/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:44227
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:46163
--webserver_port=46547
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:56.413968 17134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.390245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.130:50153)
0811 04:38:56.398215 (+ 7970us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:56.398291 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:56.399354 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:56.400142 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:56.400149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:56.400436 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:56.401095 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.401112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.402226 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.402230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.403613 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.403627 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.404468 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.404473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.404679 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.405451 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:56.405476 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:56.407181 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:56.410494 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.410502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.410519 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.410860 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.411190 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.411193 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.411195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.411418 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.411934 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:56.411940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:56.412288 (+ 348us) client_negotiation.cc:769] Sending connection context
0811 04:38:56.412500 (+ 212us) client_negotiation.cc:240] Negotiation successful
0811 04:38:56.412864 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7689,"thread_start_us":136,"threads_started":1}
I20260811 04:38:56.414273 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.389953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:50153 (local address 127.2.238.190:46163)
0811 04:38:56.390113 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:56.390118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:56.398431 (+ 8313us) server_negotiation.cc:407] Connection header received
0811 04:38:56.399644 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:56.399647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:56.399699 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:56.399744 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:56.401251 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.402062 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.403770 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.404137 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.404254 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.407410 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:56.407413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:56.407416 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:56.407441 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:56.410384 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.410982 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.410986 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.410987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.411077 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.411547 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.411550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.411552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.411767 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:56.411874 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:56.413958 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0811 04:38:56.414113 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:38:56.415444 17131 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:56.415696 17131 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.416272 17131 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
W20260811 04:38:56.417986 17131 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 04:38:56.418123 17131 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.418454 17131 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:56.419359 17131 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.419546 17131 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
W20260811 04:38:56.420532 17131 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:38:56.533878 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:38:56.534132 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:38:56.534195 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:38:56.539052 17135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:38:56.539129 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:38:56.539239 17135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:38:56.546110 17135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.238.148:54895/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:44227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=46547
--tserver_master_addrs=127.2.238.190:46163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:38:56.547235 17135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:38:56.548308 17135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:38:56.558678 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:56.558704 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:38:56.559156 17142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:38:56.559453 17135 server_base.cc:1034] running on GCE node
I20260811 04:38:56.559942 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:38:56.560689 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:38:56.561858 17135 hybrid_clock.cc:648] HybridClock initialized: now 1786423136561790 us; error 80 us; skew 500 ppm
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:56.565135 17135 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:38:56.566149 17135 webserver.cc:460] Webserver started at http://127.2.238.131:46547/ using document root <none> and password file <none>
I20260811 04:38:56.566694 17135 fs_manager.cc:359] Metadata directory not provided
I20260811 04:38:56.566797 17135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:56.587189 17135 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.003s sys 0.000s
I20260811 04:38:56.593251 17154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.594928 17135 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.000s
I20260811 04:38:56.595105 17135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0"
format_stamp: "Formatted at 2026-08-11 04:38:55 on dist-test-slave-0kls"
server_key: "01a444c20d03ec7964b73fbf1464aeca"
server_key_iv: "15a019c26e06f034874f4e8016da2e30"
server_key_version: "kuduclusterkey@0"
I20260811 04:38:56.595770 17135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:38:56.614792 17135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:38:56.618199 17135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:38:56.618420 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:38:56.619153 17135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:38:56.620159 17135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:38:56.620224 17135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.620282 17135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:38:56.620312 17135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:38:56.632030 17135 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:44227
I20260811 04:38:56.632094 17274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:44227 every 8 connection(s)
I20260811 04:38:56.633289 17135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:38:56.634780 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17135
Aug 11 04:38:56 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423136, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:38:56.650131 16987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.637410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:52375 (local address 127.2.238.190:46163)
0811 04:38:56.637721 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:56.637725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:56.637743 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:38:56.637821 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:56.637823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:56.637880 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:56.637997 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:56.639647 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.640556 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.642389 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.642793 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.642886 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.645764 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:56.645768 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:56.645771 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:56.645819 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:56.647631 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.648298 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.648303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.648305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.648402 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.648857 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.648861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.648864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.649108 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:56.649178 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:56.649907 (+ 729us) server_negotiation.cc:299] Negotiation successful
0811 04:38:56.650012 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239}
I20260811 04:38:56.651142 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.635802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46163 (local address 127.2.238.131:52375)
0811 04:38:56.636347 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:56.636402 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:56.637433 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:56.638346 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:56.638356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:56.638739 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:38:56.639412 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.639432 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.640747 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.640751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.642190 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:56.642202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.643123 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.643128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:56.643329 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.644054 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:38:56.644072 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:38:56.645595 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:38:56.647766 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.647772 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.647789 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.648157 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.648532 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:38:56.648535 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:38:56.648537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:38:56.648732 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:38:56.649277 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:38:56.649284 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:38:56.649610 (+ 326us) client_negotiation.cc:769] Sending connection context
0811 04:38:56.649848 (+ 238us) client_negotiation.cc:240] Negotiation successful
0811 04:38:56.650141 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":138,"threads_started":1}
I20260811 04:38:56.652530 17275 heartbeater.cc:347] Connected to a master server at 127.2.238.190:46163
I20260811 04:38:56.652761 17275 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.653352 17275 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:56.654433 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:56.637510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.2.238.190:46163)
0811 04:38:56.644767 (+ 7257us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:56.644772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:56.644787 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:38:56.644826 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:56.644829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:56.644874 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:56.644977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:56.645882 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.646607 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.647989 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:56.648236 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:56.648358 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:56.650492 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:56.650496 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:56.650498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:56.650523 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:56.652365 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.652923 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.652925 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.652926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.653034 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:56.653476 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:56.653485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:56.653486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:56.653663 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:56.653911 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:56.654077 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 04:38:56.654196 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7168,"thread_start_us":95,"threads_started":1}
W20260811 04:38:56.655051 17275 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:46163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260811 04:38:56.655191 17275 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.655356 17275 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:56.656121 17275 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:56.656283 17275 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
W20260811 04:38:56.656971 17275 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 11 04:38:57 dist-test-slave-0kls krb5kdc[15685](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786423137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:38:57.127182 16985 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:57.127472 16985 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:57.128697 16768 ts_manager.cc:194] Registered new tserver with Master: a1235cd6d6934ddba2a38a5ea07ebc31 (127.2.238.129:40511)
I20260811 04:38:57.130798 16768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:40623
Aug 11 04:38:57 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:57.420768 17131 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:57.421139 17131 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:57.422640 16768 ts_manager.cc:194] Registered new tserver with Master: 7c499e6e9a8e441a9d48667a32006091 (127.2.238.130:40289)
I20260811 04:38:57.423444 16768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:50153
Aug 11 04:38:57 dist-test-slave-0kls krb5kdc[15685](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423137, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for HTTP/127.2.238.148@KRBTEST.COM
I20260811 04:38:57.657173 17275 heartbeater.cc:464] Registering TS with master...
I20260811 04:38:57.657446 17275 heartbeater.cc:511] Master 127.2.238.190:46163 requested a full tablet report, sending...
I20260811 04:38:57.658279 16768 ts_manager.cc:194] Registered new tserver with Master: f48b93a8d29645d0a8e7d4b0d31f99b0 (127.2.238.131:44227)
I20260811 04:38:57.659049 16768 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:52375
I20260811 04:38:57.664951 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:38:57.680866 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.667024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32860 (local address 127.2.238.190:46163)
0811 04:38:57.667593 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.667597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.667611 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:38:57.667651 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.667654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.667701 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.667780 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:38:57.668693 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.669493 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.670629 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.670968 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.671083 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.676260 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:38:57.676265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:38:57.676267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:38:57.676295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:38:57.678217 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:57.678720 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:57.678724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:57.678726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:57.678864 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:38:57.679166 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:38:57.679181 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:38:57.679183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:38:57.679408 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:38:57.680326 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.680470 (+ 144us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.680667 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":64,"threads_started":1}
I20260811 04:38:57.685504 16768 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32860:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:38:57.690303 16768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:38:57.769483 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.752198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:44227 (local address 127.0.0.1:59514)
0811 04:38:57.754592 (+ 2394us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:57.754651 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:57.754770 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:57.756175 (+ 1405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:57.756180 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:57.756200 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:57.756330 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.757116 (+ 786us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.758486 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.758490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.761573 (+ 3083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.761584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.765224 (+ 3640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.767829 (+ 2605us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.767996 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.768062 (+ 66us) client_negotiation.cc:769] Sending connection context
0811 04:38:57.769151 (+ 1089us) client_negotiation.cc:240] Negotiation successful
0811 04:38:57.769302 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2226,"thread_start_us":125,"threads_started":1}
I20260811 04:38:57.770196 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.753479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.2.238.131:44227)
0811 04:38:57.755034 (+ 1555us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.755040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.755061 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:38:57.755127 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.755132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.755561 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.755737 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:57.757240 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.758367 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.761728 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.762609 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.762738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.762883 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.769921 (+ 7038us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.770047 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":123,"threads_started":1}
I20260811 04:38:57.770596 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.753075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.2.238.129:40511)
0811 04:38:57.753687 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.753695 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.756217 (+ 2522us) server_negotiation.cc:407] Connection header received
0811 04:38:57.756292 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.756297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.756485 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.756649 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:57.757025 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.758290 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.768494 (+ 10204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.769366 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.769471 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.769659 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.770344 (+ 685us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.770457 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":121,"threads_started":1}
I20260811 04:38:57.770138 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.752383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40511 (local address 127.0.0.1:49564)
0811 04:38:57.755961 (+ 3578us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:57.755975 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:57.756066 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:57.756793 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:57.756796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:57.756811 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:57.756897 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.756904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.764243 (+ 7339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.764247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.765380 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.765388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.769723 (+ 4335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.769727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.769828 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.769856 (+ 28us) client_negotiation.cc:769] Sending connection context
0811 04:38:57.769926 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:38:57.769997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3518,"thread_start_us":66,"threads_started":1}
I20260811 04:38:57.772213 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.752971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40289 (local address 127.0.0.1:33208)
0811 04:38:57.760492 (+ 7521us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:57.760505 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:57.760601 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:57.762939 (+ 2338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:57.762942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:57.762960 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:57.763049 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.763055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.765309 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.765312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.766505 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.766520 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.771838 (+ 5318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.771842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.771937 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.771961 (+ 24us) client_negotiation.cc:769] Sending connection context
0811 04:38:57.772025 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 04:38:57.772092 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7455,"thread_start_us":80,"threads_started":1}
I20260811 04:38:57.772753 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.753869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.2.238.130:40289)
0811 04:38:57.755346 (+ 1477us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.755352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.760714 (+ 5362us) server_negotiation.cc:407] Connection header received
0811 04:38:57.760800 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.760805 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.760988 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.761120 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:57.763178 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.765086 (+ 1908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.766674 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.767507 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.767626 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.767794 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.772483 (+ 4689us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.772601 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1390,"thread_start_us":105,"threads_started":1}
I20260811 04:38:57.774484 17186 tablet_service.cc:1467] Processing CreateTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b (DEFAULT_TABLE table=test-table [id=7a0c6dffa8f04294868dd1fe6236c375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:57.776978 17047 tablet_service.cc:1467] Processing CreateTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b (DEFAULT_TABLE table=test-table [id=7a0c6dffa8f04294868dd1fe6236c375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:57.778216 17047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4244dbf1704b43a9a6aecd384c48ae0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:57.777829 16897 tablet_service.cc:1467] Processing CreateTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b (DEFAULT_TABLE table=test-table [id=7a0c6dffa8f04294868dd1fe6236c375]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:38:57.779201 16897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4244dbf1704b43a9a6aecd384c48ae0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:57.782387 17186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4244dbf1704b43a9a6aecd384c48ae0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:38:57.793183 17315 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Bootstrap starting.
I20260811 04:38:57.796446 17315 tablet_bootstrap.cc:654] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:57.797394 17315 log.cc:824] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:57.802367 17314 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Bootstrap starting.
I20260811 04:38:57.809417 17314 tablet_bootstrap.cc:654] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:57.812075 17315 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: No bootstrap required, opened a new log
I20260811 04:38:57.812399 17315 ts_tablet_manager.cc:1397] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20260811 04:38:57.814970 17313 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Bootstrap starting.
I20260811 04:38:57.817885 17313 tablet_bootstrap.cc:654] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Neither blocks nor log segments found. Creating new log.
I20260811 04:38:57.822938 17314 log.cc:824] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:57.825126 17315 raft_consensus.cc:348] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.826524 17313 log.cc:824] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Log is configured to *not* fsync() on all Append() calls
I20260811 04:38:57.827054 17315 raft_consensus.cc:374] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:57.827203 17315 raft_consensus.cc:729] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48b93a8d29645d0a8e7d4b0d31f99b0, State: Initialized, Role: FOLLOWER
I20260811 04:38:57.827833 17315 consensus_queue.cc:260] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.829095 17315 ts_tablet_manager.cc:1428] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260811 04:38:57.830821 17275 heartbeater.cc:503] Master 127.2.238.190:46163 was elected leader, sending a full tablet report...
I20260811 04:38:57.835248 17313 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: No bootstrap required, opened a new log
I20260811 04:38:57.835538 17313 ts_tablet_manager.cc:1397] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20260811 04:38:57.836370 17314 tablet_bootstrap.cc:492] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: No bootstrap required, opened a new log
I20260811 04:38:57.836618 17314 ts_tablet_manager.cc:1397] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.000s
I20260811 04:38:57.840683 17313 raft_consensus.cc:348] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.840955 17313 raft_consensus.cc:374] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:57.841017 17313 raft_consensus.cc:729] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1235cd6d6934ddba2a38a5ea07ebc31, State: Initialized, Role: FOLLOWER
I20260811 04:38:57.841511 17313 consensus_queue.cc:260] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.842190 16985 heartbeater.cc:503] Master 127.2.238.190:46163 was elected leader, sending a full tablet report...
I20260811 04:38:57.842598 17313 ts_tablet_manager.cc:1428] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
W20260811 04:38:57.851121 16986 tablet.cc:2369] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:38:57.852959 17314 raft_consensus.cc:348] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.854557 17314 raft_consensus.cc:374] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:38:57.854934 17314 raft_consensus.cc:729] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c499e6e9a8e441a9d48667a32006091, State: Initialized, Role: FOLLOWER
I20260811 04:38:57.855549 17314 consensus_queue.cc:260] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.861963 17314 ts_tablet_manager.cc:1428] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260811 04:38:57.862649 17131 heartbeater.cc:503] Master 127.2.238.190:46163 was elected leader, sending a full tablet report...
W20260811 04:38:57.890051 17132 tablet.cc:2369] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:38:57.891366 17276 tablet.cc:2369] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:38:57.945240 17321 raft_consensus.cc:482] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:38:57.945487 17321 raft_consensus.cc:504] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.947413 17321 leader_election.cc:288] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f48b93a8d29645d0a8e7d4b0d31f99b0 (127.2.238.131:44227), 7c499e6e9a8e441a9d48667a32006091 (127.2.238.130:40289)
I20260811 04:38:57.953999 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.947811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:44227 (local address 127.2.238.129:43699)
0811 04:38:57.949351 (+ 1540us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:57.949364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:57.949509 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:57.949997 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:57.950000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:57.950020 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:57.950165 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.950172 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.951200 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.951203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.952511 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.952522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.953550 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.953554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.953645 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.953670 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:38:57.953750 (+ 80us) client_negotiation.cc:240] Negotiation successful
0811 04:38:57.953833 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"mutex_wait_us":52,"thread_start_us":148,"threads_started":1}
I20260811 04:38:57.954329 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.948397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:54725 (local address 127.2.238.130:40289)
0811 04:38:57.948530 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.948535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.950128 (+ 1593us) server_negotiation.cc:407] Connection header received
0811 04:38:57.950359 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.950362 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.950411 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.950472 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:57.950875 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.951611 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.953052 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.953602 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.953687 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.953732 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.954073 (+ 341us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.954143 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:38:57.954697 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.948255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:40289 (local address 127.2.238.129:54725)
0811 04:38:57.950025 (+ 1770us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:57.950040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:38:57.950218 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:38:57.950608 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:38:57.950611 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:38:57.950626 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:38:57.950745 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.950752 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.951750 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.951754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.952914 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:38:57.952925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.953848 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.953851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:38:57.953943 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.953963 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:38:57.954472 (+ 509us) client_negotiation.cc:240] Negotiation successful
0811 04:38:57.954544 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1703,"mutex_wait_us":30,"thread_start_us":93,"threads_started":1}
I20260811 04:38:57.954914 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.948652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:43699 (local address 127.2.238.131:44227)
0811 04:38:57.948787 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.948791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.949434 (+ 643us) server_negotiation.cc:407] Connection header received
0811 04:38:57.949645 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.949649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.949718 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:57.949767 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:38:57.950296 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.951037 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.952653 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:57.953242 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:57.953373 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:57.953426 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:57.954722 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0811 04:38:57.954800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 04:38:57.955576 17206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4244dbf1704b43a9a6aecd384c48ae0b" candidate_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" is_pre_election: true
I20260811 04:38:57.955675 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4244dbf1704b43a9a6aecd384c48ae0b" candidate_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c499e6e9a8e441a9d48667a32006091" is_pre_election: true
I20260811 04:38:57.955909 17206 raft_consensus.cc:2454] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1235cd6d6934ddba2a38a5ea07ebc31 in term 0.
I20260811 04:38:57.955907 17076 raft_consensus.cc:2454] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1235cd6d6934ddba2a38a5ea07ebc31 in term 0.
I20260811 04:38:57.956396 16861 leader_election.cc:302] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a1235cd6d6934ddba2a38a5ea07ebc31, f48b93a8d29645d0a8e7d4b0d31f99b0; no voters:
I20260811 04:38:57.956652 17321 raft_consensus.cc:2781] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:38:57.956737 17321 raft_consensus.cc:482] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:38:57.956781 17321 raft_consensus.cc:3037] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:57.958264 17321 raft_consensus.cc:504] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.958745 17321 leader_election.cc:288] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [CANDIDATE]: Term 1 election: Requested vote from peers f48b93a8d29645d0a8e7d4b0d31f99b0 (127.2.238.131:44227), 7c499e6e9a8e441a9d48667a32006091 (127.2.238.130:40289)
I20260811 04:38:57.959316 17206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4244dbf1704b43a9a6aecd384c48ae0b" candidate_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0"
I20260811 04:38:57.959430 17076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4244dbf1704b43a9a6aecd384c48ae0b" candidate_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c499e6e9a8e441a9d48667a32006091"
I20260811 04:38:57.959446 17206 raft_consensus.cc:3037] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:57.959553 17076 raft_consensus.cc:3037] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:38:57.960836 17206 raft_consensus.cc:2454] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1235cd6d6934ddba2a38a5ea07ebc31 in term 1.
I20260811 04:38:57.960892 17076 raft_consensus.cc:2454] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1235cd6d6934ddba2a38a5ea07ebc31 in term 1.
I20260811 04:38:57.961218 16861 leader_election.cc:302] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c499e6e9a8e441a9d48667a32006091, a1235cd6d6934ddba2a38a5ea07ebc31; no voters:
I20260811 04:38:57.961410 17321 raft_consensus.cc:2781] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:38:57.961679 17321 raft_consensus.cc:686] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [term 1 LEADER]: Becoming Leader. State: Replica: a1235cd6d6934ddba2a38a5ea07ebc31, State: Running, Role: LEADER
I20260811 04:38:57.962110 17321 consensus_queue.cc:237] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } }
I20260811 04:38:57.965963 16767 catalog_manager.cc:5495] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1235cd6d6934ddba2a38a5ea07ebc31 (127.2.238.129). New cstate: current_term: 1 leader_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a1235cd6d6934ddba2a38a5ea07ebc31" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40511 } health_report { overall_health: HEALTHY } } }
I20260811 04:38:58.004472 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:57.999579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.2.238.129:40511)
0811 04:38:57.999737 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0811 04:38:57.999742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:38:57.999760 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:38:57.999896 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:38:57.999899 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:38:57.999987 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:38:58.000090 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:38:58.000384 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:58.001205 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:58.002424 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:38:58.002837 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:38:58.002933 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:38:58.003347 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:38:58.003978 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:38:58.004109 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:38:58.004233 (+ 124us) server_negotiation.cc:299] Negotiation successful
0811 04:38:58.004299 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:38:58.012315 17076 raft_consensus.cc:1261] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 1 FOLLOWER]: Refusing update from remote peer a1235cd6d6934ddba2a38a5ea07ebc31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:38:58.012380 17206 raft_consensus.cc:1261] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 1 FOLLOWER]: Refusing update from remote peer a1235cd6d6934ddba2a38a5ea07ebc31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:38:58.013038 17321 consensus_queue.cc:1035] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f48b93a8d29645d0a8e7d4b0d31f99b0" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:38:58.013247 17328 consensus_queue.cc:1035] T 4244dbf1704b43a9a6aecd384c48ae0b P a1235cd6d6934ddba2a38a5ea07ebc31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c499e6e9a8e441a9d48667a32006091" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 40289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:38:58.023011 17338 mvcc.cc:205] Tried to move back new op lower bound from 7317189173291184128 to 7317189173095890944. Current Snapshot: MvccSnapshot[applied={T|T < 7317189173291184128}]
I20260811 04:38:58.026623 17337 mvcc.cc:205] Tried to move back new op lower bound from 7317189173291184128 to 7317189173095890944. Current Snapshot: MvccSnapshot[applied={T|T < 7317189173291184128}]
I20260811 04:38:58.032804 16768 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:38:58.032984 16768 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32860:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:38:58.035605 16768 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11595274755d472eba945c3e6acb3505: Sending DeleteTablet for 3 replicas of tablet 4244dbf1704b43a9a6aecd384c48ae0b
I20260811 04:38:58.036335 17186 tablet_service.cc:1514] Processing DeleteTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:58 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:59514
I20260811 04:38:58.036577 17047 tablet_service.cc:1514] Processing DeleteTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:58 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:33208
I20260811 04:38:58.036880 16897 tablet_service.cc:1514] Processing DeleteTablet for tablet 4244dbf1704b43a9a6aecd384c48ae0b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:38:58 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:49564
I20260811 04:38:58.037180 17346 tablet_replica.cc:330] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: stopping tablet replica
I20260811 04:38:58.037514 17346 raft_consensus.cc:2229] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:38:58.037940 17346 raft_consensus.cc:2258] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:38:58.038295 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16811
I20260811 04:38:58.039016 17345 tablet_replica.cc:330] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: stopping tablet replica
I20260811 04:38:58.039395 17345 raft_consensus.cc:2229] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:38:58.039664 17346 ts_tablet_manager.cc:1905] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:38:58.039680 17345 raft_consensus.cc:2258] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:38:58.041297 17345 ts_tablet_manager.cc:1905] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:38:58.042574 17346 ts_tablet_manager.cc:1918] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:38:58.042737 17346 log.cc:1196] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/4244dbf1704b43a9a6aecd384c48ae0b
I20260811 04:38:58.043129 17346 ts_tablet_manager.cc:1939] T 4244dbf1704b43a9a6aecd384c48ae0b P 7c499e6e9a8e441a9d48667a32006091: Deleting consensus metadata
I20260811 04:38:58.044384 16751 catalog_manager.cc:4841] TS 7c499e6e9a8e441a9d48667a32006091 (127.2.238.130:40289): tablet 4244dbf1704b43a9a6aecd384c48ae0b (table test-table [id=7a0c6dffa8f04294868dd1fe6236c375]) successfully deleted
I20260811 04:38:58.044512 17345 ts_tablet_manager.cc:1918] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:38:58.044597 17345 log.cc:1196] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/4244dbf1704b43a9a6aecd384c48ae0b
I20260811 04:38:58.044906 17345 ts_tablet_manager.cc:1939] T 4244dbf1704b43a9a6aecd384c48ae0b P f48b93a8d29645d0a8e7d4b0d31f99b0: Deleting consensus metadata
I20260811 04:38:58.046041 16751 catalog_manager.cc:4841] TS f48b93a8d29645d0a8e7d4b0d31f99b0 (127.2.238.131:44227): tablet 4244dbf1704b43a9a6aecd384c48ae0b (table test-table [id=7a0c6dffa8f04294868dd1fe6236c375]) successfully deleted
W20260811 04:38:58.050273 16754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:40511 (error 108)
I20260811 04:38:58.051014 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16989
I20260811 04:38:58.051301 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:38:58.051018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40511 (local address 127.0.0.1:49580)
0811 04:38:58.051148 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:38:58.051218 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 04:38:58.054502 16754 catalog_manager.cc:4584] TS a1235cd6d6934ddba2a38a5ea07ebc31 (127.2.238.129:40511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4244dbf1704b43a9a6aecd384c48ae0b: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40511: connect: Connection refused (error 111)
I20260811 04:38:58.058043 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17135
I20260811 04:38:58.067359 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 16732
I20260811 04:38:58.076642 3002 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260811 04:38:58.421731 3002 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260811 04:38:58.421914 3002 mini_ranger.cc:66] Stopping Ranger...
I20260811 04:38:58.766542 3002 mini_ranger.cc:68] Stopped Ranger
2026-08-11 04:38:58.766 UTC [15718] LOG: received smart shutdown request
2026-08-11 04:38:58.771 UTC [15718] LOG: background worker "logical replication launcher" (PID 15727) exited with exit code 1
2026-08-11 04:38:58.771 UTC [15722] LOG: shutting down
2026-08-11 04:38:58.930 UTC [15718] LOG: database system is shut down
2026-08-11T04:38:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47038 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:38:59 dist-test-slave-0kls krb5kdc[17358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:38:59 dist-test-slave-0kls krb5kdc[17358](info): set up 2 sockets
Aug 11 04:38:59 dist-test-slave-0kls krb5kdc[17358](info): commencing operation
krb5kdc: starting...
W20260811 04:39:00.105947 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:39:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:39:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:39:00.225003 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46079
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:00.335999 17371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:00.336253 17371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:00.336292 17371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:00.340396 17371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:00.340461 17371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:00.340481 17371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:00.340498 17371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:00.340513 17371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:00.345534 17371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:00.346606 17371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:00.347535 17371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:00.356052 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:00.356240 17371 server_base.cc:1034] running on GCE node
W20260811 04:39:00.356056 17376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:00.356055 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:00.356937 17371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:00.358112 17371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:00.359297 17371 hybrid_clock.cc:648] HybridClock initialized: now 1786423140359284 us; error 25 us; skew 500 ppm
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:00.362066 17371 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:00.362929 17371 webserver.cc:460] Webserver started at http://127.2.238.190:44295/ using document root <none> and password file <none>
I20260811 04:39:00.363438 17371 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:00.363495 17371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:00.363660 17371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:00.365634 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "29c2574b1a7549188c97eaf214fc6979"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "7482ce972d7ebabddb24c24f2d91dcdf"
server_key_iv: "50f89fa26443b7d065560c12eac290ff"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.366237 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "29c2574b1a7549188c97eaf214fc6979"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "7482ce972d7ebabddb24c24f2d91dcdf"
server_key_iv: "50f89fa26443b7d065560c12eac290ff"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.369874 17371 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:39:00.372116 17387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:00.373279 17371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:39:00.373399 17371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "29c2574b1a7549188c97eaf214fc6979"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "7482ce972d7ebabddb24c24f2d91dcdf"
server_key_iv: "50f89fa26443b7d065560c12eac290ff"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.373499 17371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:00.388787 17371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:00.391613 17371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:00.391812 17371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:00.398902 17371 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:46079
I20260811 04:39:00.398929 17441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:46079 every 8 connection(s)
I20260811 04:39:00.399879 17371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:00.402050 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17371
I20260811 04:39:00.402187 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:39:00.402472 3002 external_mini_cluster.cc:1428] Setting key 5ea8e4bd07549097f10ee86507bbf6f5
I20260811 04:39:00.402747 17442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:00.406497 17442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.189@KRBTEST.COM" created.
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:00.421694 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.408791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46079 (local address 127.0.0.1:45734)
0811 04:39:00.409387 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.409434 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.410214 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.410864 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.410868 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.411229 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.411472 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.411486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.412564 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.412567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.413389 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.413400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.414141 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.414145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.414322 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.414926 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.414941 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.416480 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.418759 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.418768 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.418783 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.419007 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.419409 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.419411 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.419413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.419519 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.420315 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.420320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.420443 (+ 123us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.420582 (+ 139us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.420846 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":3,"spinlock_wait_cycles":15744,"thread_start_us":143,"threads_started":1}
I20260811 04:39:00.421700 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.408789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45734 (local address 127.2.238.190:46079)
0811 04:39:00.409211 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.409230 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.409533 (+ 303us) server_negotiation.cc:407] Connection header received
0811 04:39:00.410505 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.410509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.410668 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.410869 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.411613 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.412445 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.413550 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.413979 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.414159 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.416646 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.416655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.416671 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.416701 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.418647 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.419126 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.419135 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.419139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.419291 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.419619 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.419623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.419627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.420197 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.420342 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.420713 (+ 371us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.420929 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":277,"thread_start_us":138,"threads_started":1}
I20260811 04:39:00.424139 17442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.424852 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.424581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38300)
0811 04:39:00.424710 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.424796 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 04:39:00.425073 17389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
I20260811 04:39:00.426048 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.425778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38308)
0811 04:39:00.425920 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.425986 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 04:39:00.426205 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:42015: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
I20260811 04:39:00.430656 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:42015
--webserver_interface=127.2.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:00.475446 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } attempt: 1
I20260811 04:39:00.476475 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.476168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38312)
0811 04:39:00.476330 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.476395 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 04:39:00.477298 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.477111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38318)
0811 04:39:00.477205 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.477247 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 04:39:00.477438 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:42015: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
I20260811 04:39:00.545598 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } attempt: 2
I20260811 04:39:00.546454 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.546171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38330)
0811 04:39:00.546320 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.546381 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 04:39:00.546517 17449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:00.546810 17449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:00.546914 17449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 04:39:00.547295 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.547072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38332)
0811 04:39:00.547185 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.547234 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:39:00.547422 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:42015: Network error: Client connection negotiation failed: client connection to 127.2.238.189:42015: connect: Connection refused (error 111)
W20260811 04:39:00.550884 17449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:00.550930 17449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:00.550946 17449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:00.550979 17449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:00.551009 17449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:00.555883 17449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:42015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:00.556788 17449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:00.557704 17449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:00.565984 17455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:00.565990 17458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:00.565990 17456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:00.566264 17449 server_base.cc:1034] running on GCE node
I20260811 04:39:00.566949 17449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:00.567996 17449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:00.569155 17449 hybrid_clock.cc:648] HybridClock initialized: now 1786423140569129 us; error 39 us; skew 500 ppm
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:00.571843 17449 init.cc:377] Logged in from keytab as kudu/127.2.238.189@KRBTEST.COM (short username kudu)
I20260811 04:39:00.572626 17449 webserver.cc:460] Webserver started at http://127.2.238.189:39005/ using document root <none> and password file <none>
I20260811 04:39:00.573108 17449 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:00.573161 17449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:00.573315 17449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:00.575331 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data/instance:
uuid: "b67933f249de4548b985ecece0b93985"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "c264abe22088a0fe9353357f6a9b82bb"
server_key_iv: "ae04b1bced88e53e7c4b2fb62c5d8842"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.575822 17449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal/instance:
uuid: "b67933f249de4548b985ecece0b93985"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "c264abe22088a0fe9353357f6a9b82bb"
server_key_iv: "ae04b1bced88e53e7c4b2fb62c5d8842"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.580279 17449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260811 04:39:00.582772 17465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:00.584039 17449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:00.584190 17449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "b67933f249de4548b985ecece0b93985"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "c264abe22088a0fe9353357f6a9b82bb"
server_key_iv: "ae04b1bced88e53e7c4b2fb62c5d8842"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.584295 17449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:00.609894 17449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:00.612717 17449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:00.612927 17449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:00.620151 17449 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.189:42015
I20260811 04:39:00.620194 17519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.189:42015 every 8 connection(s)
I20260811 04:39:00.621091 17449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
I20260811 04:39:00.623900 17520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:00.628084 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17449
I20260811 04:39:00.628226 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-1/wal/instance
I20260811 04:39:00.627961 17520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.628567 3002 external_mini_cluster.cc:1428] Setting key e84e81c80aa28ad4b9791f5540b1a891
WARNING: no policy specified for kudu/127.2.238.188@KRBTEST.COM; defaulting to no policy
I20260811 04:39:00.638556 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.188@KRBTEST.COM" created.
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:39:00.646078 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.630358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.2.238.190:46079)
0811 04:39:00.630536 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.630544 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.631470 (+ 926us) server_negotiation.cc:407] Connection header received
0811 04:39:00.632440 (+ 970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.632443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.632510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.632614 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.633773 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.634369 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.635493 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.635728 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.635843 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.641270 (+ 5427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.641274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.641276 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.641298 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.643694 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.644202 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.644209 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.644212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.644314 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.644751 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.644755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.644757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.644989 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.645098 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.645846 (+ 748us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.645960 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:39:00.647286 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.630798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46079 (local address 127.0.0.1:45744)
0811 04:39:00.631317 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.631361 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.632212 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.632790 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.632797 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.633131 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.633618 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.633631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.634516 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.634519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.635360 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.635369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.635912 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.635916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.636084 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.636731 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.636747 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.641116 (+ 4369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.643816 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.643820 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.643832 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.644085 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.644423 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.644429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.644434 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.644635 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.645123 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.645132 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.645473 (+ 341us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.645754 (+ 281us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.646131 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":152,"threads_started":1}
I20260811 04:39:00.648732 17520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.649637 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.639085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38338 (local address 127.2.238.189:42015)
0811 04:39:00.639367 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.639377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.639398 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:39:00.639432 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.639436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.639585 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.639711 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.640208 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.641287 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.642398 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.642994 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.643113 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.645209 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.645219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.645233 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.645264 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.647585 (+ 2321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.648054 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.648060 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.648062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.648149 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.648468 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.648472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.648475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.648798 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.648979 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.649306 (+ 327us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.649454 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":105,"threads_started":1}
I20260811 04:39:00.649740 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.638982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38338)
0811 04:39:00.639109 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.639126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.639199 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.639737 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.639739 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.639901 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.640091 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.640098 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.641420 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.641423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.642259 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.642273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.643124 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.643130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.643259 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.643713 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.643732 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.645101 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.647700 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.647705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.647707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.647939 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.648256 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.648259 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.648261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.648364 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.648946 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.648949 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.649052 (+ 103us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.649190 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.649328 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 04:39:00.651575 17442 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.652261 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.651989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:36970)
0811 04:39:00.652119 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.652175 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260811 04:39:00.652990 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.652681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:36984)
0811 04:39:00.652865 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.652914 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260811 04:39:00.653334 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:39:00.657941 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.649254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38352)
0811 04:39:00.649518 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.649534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.649614 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.650214 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.650219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.650363 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.650452 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.650457 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.651345 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.651351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.652162 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.652173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.652717 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.652720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.652866 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.653352 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.653363 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.654658 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.656347 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.656357 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.656358 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.656540 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.656850 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.656855 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.656858 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.657031 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.657401 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.657404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.657506 (+ 102us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.657637 (+ 131us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.657754 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":2}
I20260811 04:39:00.658119 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.649481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.2.238.189:42015)
0811 04:39:00.649981 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.649986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.650003 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:39:00.650040 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.650044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.650086 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.650162 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.650563 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.651226 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.652289 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.652611 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.652706 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.654787 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.654790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.654791 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.654811 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.656242 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.656645 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.656648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.656649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.656718 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.657145 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.657147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.657148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.657319 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.657392 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.657697 (+ 305us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.657896 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":438,"thread_start_us":85,"threads_started":1}
I20260811 04:39:00.658814 17520 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.659471 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.659251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:36994)
0811 04:39:00.659345 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.659411 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
W20260811 04:39:00.660144 17466 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
I20260811 04:39:00.660866 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:46607
--webserver_interface=127.2.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:37393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:00.661126 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.660851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37008)
0811 04:39:00.660999 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.661061 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260811 04:39:00.661332 17520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
I20260811 04:39:00.684618 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 1
I20260811 04:39:00.685552 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.685191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37024)
0811 04:39:00.685378 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.685456 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260811 04:39:00.686345 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.686085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37038)
0811 04:39:00.686212 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.686267 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 04:39:00.686501 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
I20260811 04:39:00.710602 17520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 1
I20260811 04:39:00.711557 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.711271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37048)
0811 04:39:00.711426 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.711484 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 04:39:00.712400 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.712171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37060)
0811 04:39:00.712288 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.712343 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 04:39:00.712529 17520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
I20260811 04:39:00.761691 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 2
I20260811 04:39:00.762704 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.762321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37072)
0811 04:39:00.762522 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.762600 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260811 04:39:00.763502 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.763244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37086)
0811 04:39:00.763372 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.763430 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260811 04:39:00.763669 17442 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
W20260811 04:39:00.775998 17528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:00.776268 17528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:00.776307 17528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:00.780620 17528 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:00.780687 17528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:00.780707 17528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:00.780725 17528 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:00.780740 17528 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:00.780752 17520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 2
I20260811 04:39:00.781653 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.781320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37090)
0811 04:39:00.781478 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.781549 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260811 04:39:00.782544 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.782324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37100)
0811 04:39:00.782444 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.782491 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 04:39:00.782691 17520 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:46607: Network error: Client connection negotiation failed: client connection to 127.2.238.188:46607: connect: Connection refused (error 111)
I20260811 04:39:00.786574 17528 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46079,127.2.238.189:42015,127.2.238.188:46607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:00.787698 17528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:00.788859 17528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:00.797405 17534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:00.797456 17533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:00.797456 17537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:00.797580 17528 server_base.cc:1034] running on GCE node
I20260811 04:39:00.798355 17528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:00.799463 17528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:00.800655 17528 hybrid_clock.cc:648] HybridClock initialized: now 1786423140800639 us; error 43 us; skew 500 ppm
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:00.805171 17528 init.cc:377] Logged in from keytab as kudu/127.2.238.188@KRBTEST.COM (short username kudu)
I20260811 04:39:00.806571 17528 webserver.cc:460] Webserver started at http://127.2.238.188:41015/ using document root <none> and password file <none>
I20260811 04:39:00.807267 17528 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:00.807399 17528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:00.807617 17528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:00.810103 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data/instance:
uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "2a3ab0603a0f9250e4f72b7e3c351ff3"
server_key_iv: "6d179476af1c5799a50d0b714c847c76"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.810796 17528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal/instance:
uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "2a3ab0603a0f9250e4f72b7e3c351ff3"
server_key_iv: "6d179476af1c5799a50d0b714c847c76"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.814392 17528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:00.816625 17544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:00.817829 17528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260811 04:39:00.817965 17528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5"
format_stamp: "Formatted at 2026-08-11 04:39:00 on dist-test-slave-0kls"
server_key: "2a3ab0603a0f9250e4f72b7e3c351ff3"
server_key_iv: "6d179476af1c5799a50d0b714c847c76"
server_key_version: "encryptionkey@0"
I20260811 04:39:00.818065 17528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:00.835023 17528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:00.838433 17528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:00.838630 17528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:00.845780 17528 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.188:46607
I20260811 04:39:00.845845 17598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.188:46607 every 8 connection(s)
I20260811 04:39:00.846732 17528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
I20260811 04:39:00.847697 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17528
I20260811 04:39:00.847836 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786422982503306-3002-0/minicluster-data/master-2/wal/instance
I20260811 04:39:00.848172 3002 external_mini_cluster.cc:1428] Setting key 00109a4a1025b87acedd0154161f35d9
I20260811 04:39:00.850147 17599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:00.853713 17599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:00.862635 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.849605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37108 (local address 127.2.238.188:46607)
0811 04:39:00.850051 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.850066 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.850100 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:39:00.850792 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.850808 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.851123 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.851456 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.852308 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.853225 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.854149 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.854843 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.855013 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.857646 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.857651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.857660 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.857686 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.859539 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.859999 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.860005 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.860029 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.860187 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.860483 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.860486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.860488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.860928 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.861097 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.861423 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.861733 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":151,"threads_started":1}
I20260811 04:39:00.862835 17442 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 3
I20260811 04:39:00.865295 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.855695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.2.238.190:46079)
0811 04:39:00.855851 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.855857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.856119 (+ 262us) server_negotiation.cc:407] Connection header received
0811 04:39:00.856277 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.856280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.856334 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.856422 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.857104 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.857645 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.858577 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.858806 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.858916 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.861211 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.861214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.861216 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.861260 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.863214 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.863724 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.863728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.863730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.863827 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.864142 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.864145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.864147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.864426 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.864509 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.864970 (+ 461us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.865117 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":46}
I20260811 04:39:00.865856 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.855633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:46079 (local address 127.0.0.1:45748)
0811 04:39:00.855983 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.856021 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.856171 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.856472 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.856478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.856848 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.856970 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.856976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.857774 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.857778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.858449 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.858459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.858928 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.858932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.859056 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.859536 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.859548 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.861055 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.863352 (+ 2297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.863360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.863364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.863597 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.863946 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.863949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.863951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.864039 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.864597 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.864602 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.864737 (+ 135us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.865438 (+ 701us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.865666 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
I20260811 04:39:00.866976 17599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:00.872958 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.863432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.2.238.188:46607)
0811 04:39:00.863790 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.863794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.863809 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:39:00.863871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.863873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.863908 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.863999 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.864515 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.865186 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.866362 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.866620 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.866727 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.868759 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.868763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.868765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.868788 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.870792 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.871272 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.871276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.871277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.871357 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.871706 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.871708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.871710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.871926 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.872115 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.872668 (+ 553us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.872802 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:39:00.873232 17446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.863255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37122)
0811 04:39:00.863690 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.863706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.863784 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.864014 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.864017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.864217 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.864366 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.864372 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.865324 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.865330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.866223 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.866234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.866739 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.866743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.866826 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.867246 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.867261 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.868633 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.870898 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.870901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.870903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.871150 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.871466 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.871469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.871471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.871560 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.872083 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.872086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.872188 (+ 102us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.872597 (+ 409us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.872771 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":68}
I20260811 04:39:00.873821 17520 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } attempt: 3
I20260811 04:39:00.876433 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.865386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.2.238.190:46079)
0811 04:39:00.865910 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.865915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.865931 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:39:00.865968 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.865971 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.866009 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.866083 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.867232 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.867908 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.869075 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.869394 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.869492 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.872058 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.872064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.872069 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.872094 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.874138 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.874912 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.874916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.874918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.875005 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.875336 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.875342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.875345 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.875575 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.875753 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.876159 (+ 406us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.876275 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":448,"thread_start_us":90,"threads_started":1}
I20260811 04:39:00.876892 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: Bootstrap starting.
I20260811 04:39:00.877205 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.867472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:42015 (local address 127.0.0.1:38364)
0811 04:39:00.867614 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.867626 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.867688 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.868093 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.868095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.868272 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.868782 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.868791 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.869862 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.869867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.870676 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.870689 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.871430 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.871436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.871537 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.871921 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.871934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.873335 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.875536 (+ 2201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.875539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.875541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.875768 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.876105 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.876107 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.876109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.876191 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.876731 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.876737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.876833 (+ 96us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.876939 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.877062 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260811 04:39:00.877247 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.867571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38364 (local address 127.2.238.189:42015)
0811 04:39:00.867858 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.867863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.867877 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:39:00.867920 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.867923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.867966 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.868057 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.868923 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.869673 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.870823 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.871206 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.871283 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.873471 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.873477 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.873481 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.873517 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.875419 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.875897 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.875900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.875902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.875986 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.876315 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.876318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.876320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.876574 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.876655 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.877010 (+ 355us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.877114 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183}
I20260811 04:39:00.878037 17599 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:39:00.880000 17442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: Neither blocks nor log segments found. Creating new log.
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:39:00.880954 17442 log.cc:824] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:00.883592 17442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: No bootstrap required, opened a new log
I20260811 04:39:00.884413 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.874249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37138)
0811 04:39:00.874404 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.874418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.874487 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.874917 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.874921 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.875141 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.875228 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.875234 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.876179 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.876183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.876883 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.876893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.878820 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.878827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.878924 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.879398 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.879412 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.880792 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.882811 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.882816 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.882821 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.883045 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.883343 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.883346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.883348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.883432 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.883933 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.883936 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.884040 (+ 104us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.884138 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.884256 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260811 04:39:00.884971 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.874301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.2.238.188:46607)
0811 04:39:00.874670 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.874677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.874692 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:39:00.874727 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.874732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.874777 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.875329 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.875406 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.876031 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.877985 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.878655 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.878796 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.880933 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.880936 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.880938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.880966 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.882676 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.883157 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.883160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.883161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.883240 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.883534 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.883537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.883538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.883786 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.883883 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.884717 (+ 834us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.884814 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":301}
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:39:00.887987 17520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: Bootstrap starting.
I20260811 04:39:00.887882 17442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.888111 17442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 11 04:39:00 dist-test-slave-0kls krb5kdc[17358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423140, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:39:00.888159 17442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c2574b1a7549188c97eaf214fc6979, State: Initialized, Role: FOLLOWER
I20260811 04:39:00.888712 17442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.890112 17442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:00.890313 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.878992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38370 (local address 127.2.238.189:42015)
0811 04:39:00.879099 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.879103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.879541 (+ 438us) server_negotiation.cc:407] Connection header received
0811 04:39:00.879636 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.879639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.879681 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.879754 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.880614 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.881333 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.882412 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.882752 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.882871 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.885708 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.885711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.885714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.885736 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.887659 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.888223 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.888229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.888231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.888335 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.888717 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.888720 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.888722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.888962 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.889155 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.889937 (+ 782us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.890102 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 04:39:00.890851 17520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:00.891782 17520 log.cc:824] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:00.893952 17520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: No bootstrap required, opened a new log
I20260811 04:39:00.894037 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260811 04:39:00.896045 17620 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:39:00.896127 17620 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260811 04:39:00.897935 17393 tablet.cc:2369] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:00.898557 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.878572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:46607 (local address 127.0.0.1:37144)
0811 04:39:00.879705 (+ 1133us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:00.879719 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:00.879781 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:00.881999 (+ 2218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:00.882002 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:00.882244 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:00.882359 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.882367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.883593 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.883597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.884809 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:00.884824 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.885851 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.885858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:00.886248 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.886936 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:00.886971 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:00.888756 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:00.891672 (+ 2916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.891675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.891677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.892008 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.892362 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:00.892365 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:00.892367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:00.892461 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:00.898056 (+ 5595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:00.898059 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:00.898156 (+ 97us) client_negotiation.cc:769] Sending connection context
0811 04:39:00.898262 (+ 106us) client_negotiation.cc:240] Negotiation successful
0811 04:39:00.898396 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":1}
I20260811 04:39:00.895627 17371 master_runner.cc:186] Error getting master registration for 127.2.238.189:42015: OK, instance_id { permanent_uuid: "b67933f249de4548b985ecece0b93985" instance_seqno: 1786423140618112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:39:00.898607 17520 raft_consensus.cc:348] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.898841 17371 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 04:39:00.898840 17520 raft_consensus.cc:374] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:00.898906 17520 raft_consensus.cc:729] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b67933f249de4548b985ecece0b93985, State: Initialized, Role: FOLLOWER
I20260811 04:39:00.899153 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.878767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.2.238.188:46607)
0811 04:39:00.881470 (+ 2703us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.881481 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.881498 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:39:00.881535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.881540 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.881597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.881690 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.882471 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.883192 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.884961 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.885608 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.885684 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.889017 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.889024 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.889030 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.889073 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.891559 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.892145 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.892151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.892153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.892242 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.892560 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.892562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.892564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.892776 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.898015 (+ 5239us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.898835 (+ 820us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.898998 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2622}
I20260811 04:39:00.899353 17520 consensus_queue.cc:260] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.900674 17520 sys_catalog.cc:564] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:00.901973 17607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:00.902103 17607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:00.902314 17599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: Bootstrap starting.
I20260811 04:39:00.902966 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:00.903100 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:00.904928 17599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:00.906088 17599 log.cc:824] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: Log is configured to *not* fsync() on all Append() calls
W20260811 04:39:00.905921 17638 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:39:00.906795 17638 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:39:00.913160 17599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: No bootstrap required, opened a new log
I20260811 04:39:00.910988 17449 master_runner.cc:186] Error getting master registration for 127.2.238.188:46607: OK, instance_id { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" instance_seqno: 1786423140843767 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:39:00.915060 17449 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 04:39:00.916587 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.903475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45776 (local address 127.2.238.190:46079)
0811 04:39:00.903609 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.903614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.907149 (+ 3535us) server_negotiation.cc:407] Connection header received
0811 04:39:00.907355 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.907358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.907402 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.907451 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.908456 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.909255 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.910413 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.910695 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.910774 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.912672 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.912677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.912679 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.912707 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.914553 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.915122 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.915126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.915128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.915217 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.915595 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.915598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.915600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.915811 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.915935 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.916319 (+ 384us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.916431 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 04:39:00.917223 17599 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.917483 17599 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:00.917551 17599 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb7efd5e5114ecab8a9c2ccb43addf5, State: Initialized, Role: FOLLOWER
I20260811 04:39:00.918099 17599 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:00.919498 17599 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:00.928540 17528 master_runner.cc:263] No leader master found from master dcb7efd5e5114ecab8a9c2ccb43addf5
I20260811 04:39:00.928607 17528 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 04:39:00.929409 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.906981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37160 (local address 127.2.238.188:46607)
0811 04:39:00.907107 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.907112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.917389 (+ 10277us) server_negotiation.cc:407] Connection header received
0811 04:39:00.917742 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.917745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.917815 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.917867 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.918910 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.919734 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.921037 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.921346 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.921418 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.923204 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.923208 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.923211 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.923234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.925078 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.925617 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.925620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.925622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.925719 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.926044 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.926050 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.926052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.926283 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.927437 (+ 1154us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.929128 (+ 1691us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.929254 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 04:39:00.929625 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:00.906552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38374 (local address 127.2.238.189:42015)
0811 04:39:00.906653 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:00.906657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:00.910060 (+ 3403us) server_negotiation.cc:407] Connection header received
0811 04:39:00.910702 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:00.910705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:00.910746 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:00.910790 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:00.921635 (+ 10845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.922366 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.923470 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:00.923760 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:00.923833 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:00.926049 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:00.926052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:00.926055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:00.926088 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:00.927821 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.928321 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.928324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.928326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.928407 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:00.928701 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:00.928704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:00.928706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:00.928907 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:00.929020 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:00.929354 (+ 334us) server_negotiation.cc:299] Negotiation successful
0811 04:39:00.929481 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 04:39:00.929951 17645 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:00.930069 17645 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: This master's current role is: FOLLOWER
W20260811 04:39:00.933375 17657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:39:00.933450 17657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260811 04:39:01.089987 17550 tablet.cc:2369] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:39:01.114549 17471 tablet.cc:2369] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:01.200384 17607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:01.200616 17607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:01.201771 17607 leader_election.cc:288] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b67933f249de4548b985ecece0b93985 (127.2.238.189:42015), dcb7efd5e5114ecab8a9c2ccb43addf5 (127.2.238.188:46607)
I20260811 04:39:01.202337 17492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29c2574b1a7549188c97eaf214fc6979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67933f249de4548b985ecece0b93985" is_pre_election: true
I20260811 04:39:01.202420 17571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29c2574b1a7549188c97eaf214fc6979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" is_pre_election: true
I20260811 04:39:01.202695 17492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c2574b1a7549188c97eaf214fc6979 in term 0.
I20260811 04:39:01.202860 17571 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c2574b1a7549188c97eaf214fc6979 in term 0.
I20260811 04:39:01.203191 17389 leader_election.cc:302] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c2574b1a7549188c97eaf214fc6979, b67933f249de4548b985ecece0b93985; no voters:
I20260811 04:39:01.203498 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:39:01.203595 17607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:39:01.203651 17607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:01.204753 17607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:01.205147 17607 leader_election.cc:288] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [CANDIDATE]: Term 1 election: Requested vote from peers b67933f249de4548b985ecece0b93985 (127.2.238.189:42015), dcb7efd5e5114ecab8a9c2ccb43addf5 (127.2.238.188:46607)
I20260811 04:39:01.205466 17571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29c2574b1a7549188c97eaf214fc6979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5"
I20260811 04:39:01.205466 17492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "29c2574b1a7549188c97eaf214fc6979" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b67933f249de4548b985ecece0b93985"
I20260811 04:39:01.205602 17492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:01.205602 17571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:01.206930 17492 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c2574b1a7549188c97eaf214fc6979 in term 1.
I20260811 04:39:01.206930 17571 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c2574b1a7549188c97eaf214fc6979 in term 1.
I20260811 04:39:01.207278 17388 leader_election.cc:302] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c2574b1a7549188c97eaf214fc6979, dcb7efd5e5114ecab8a9c2ccb43addf5; no voters:
I20260811 04:39:01.207436 17607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:01.207612 17607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [term 1 LEADER]: Becoming Leader. State: Replica: 29c2574b1a7549188c97eaf214fc6979, State: Running, Role: LEADER
I20260811 04:39:01.207944 17607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } }
I20260811 04:39:01.210213 17660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c2574b1a7549188c97eaf214fc6979. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.210331 17660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:01.210669 17662 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:01.211464 17662 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:01.216243 17571 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [term 1 FOLLOWER]: Refusing update from remote peer 29c2574b1a7549188c97eaf214fc6979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:01.216243 17492 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [term 1 FOLLOWER]: Refusing update from remote peer 29c2574b1a7549188c97eaf214fc6979: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:01.216823 17607 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:01.217017 17660 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:01.220721 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c2574b1a7549188c97eaf214fc6979. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.220722 17645 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29c2574b1a7549188c97eaf214fc6979. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.220834 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:01.220832 17645 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:01.223738 17626 sys_catalog.cc:455] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.223707 17660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.223737 17607 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.223826 17626 sys_catalog.cc:458] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:01.223826 17660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:01.223835 17607 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:01.223946 17645 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "29c2574b1a7549188c97eaf214fc6979" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c2574b1a7549188c97eaf214fc6979" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46079 } } peers { permanent_uuid: "b67933f249de4548b985ecece0b93985" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 42015 } } peers { permanent_uuid: "dcb7efd5e5114ecab8a9c2ccb43addf5" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 46607 } } }
I20260811 04:39:01.224066 17645 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5 [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:39:01.224089 17665 mvcc.cc:205] Tried to move back new op lower bound from 7317189186416050176 to 7317189186390192128. Current Snapshot: MvccSnapshot[applied={T|T < 7317189186416050176}]
I20260811 04:39:01.225538 17662 catalog_manager.cc:1347] Generated new cluster ID: 24fb45ccaca2450c9199a5d8a7c3e759
I20260811 04:39:01.225618 17662 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:01.245215 17662 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:39:01.246456 17662 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:01.258088 17662 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29c2574b1a7549188c97eaf214fc6979: Generated new TSK 0
I20260811 04:39:01.259090 17662 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:39:01.908157 17638 catalog_manager.cc:1259] Loaded cluster ID: 24fb45ccaca2450c9199a5d8a7c3e759
I20260811 04:39:01.908241 17638 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: loading cluster ID for follower catalog manager: success
I20260811 04:39:01.909940 17638 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: acquiring CA information for follower catalog manager: success
I20260811 04:39:01.911015 17638 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b67933f249de4548b985ecece0b93985: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:39:01.934834 17657 catalog_manager.cc:1259] Loaded cluster ID: 24fb45ccaca2450c9199a5d8a7c3e759
I20260811 04:39:01.934949 17657 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: loading cluster ID for follower catalog manager: success
I20260811 04:39:01.936565 17657 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: acquiring CA information for follower catalog manager: success
I20260811 04:39:01.937709 17657 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcb7efd5e5114ecab8a9c2ccb43addf5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:39:02.336512 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17371
I20260811 04:39:02.346109 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17449
I20260811 04:39:02.353964 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17528
2026-08-11T04:39:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3301 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:39:02 dist-test-slave-0kls krb5kdc[17678](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:39:02 dist-test-slave-0kls krb5kdc[17678](info): set up 2 sockets
Aug 11 04:39:02 dist-test-slave-0kls krb5kdc[17678](info): commencing operation
krb5kdc: starting...
W20260811 04:39:03.389060 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:39:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:39:03.516719 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46097
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32881
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:46097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:03.628639 17691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:03.628939 17691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:03.629020 17691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:03.632944 17691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:03.633031 17691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:03.633066 17691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:03.633114 17691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:03.633167 17691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:03.638345 17691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32881
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:46097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:46097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:03.639431 17691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:03.640535 17691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:03.649065 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:03.649264 17691 server_base.cc:1034] running on GCE node
W20260811 04:39:03.649096 17696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:03.649065 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:03.650009 17691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:03.650991 17691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:03.652145 17691 hybrid_clock.cc:648] HybridClock initialized: now 1786423143652130 us; error 28 us; skew 500 ppm
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17678](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423143, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:03.654763 17691 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:03.655628 17691 webserver.cc:460] Webserver started at http://127.2.238.190:43495/ using document root <none> and password file <none>
I20260811 04:39:03.656193 17691 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:03.656256 17691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:03.656507 17691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:03.658600 17691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "94601691df0d41bbad9fe4e1ad3ffc6c"
format_stamp: "Formatted at 2026-08-11 04:39:03 on dist-test-slave-0kls"
server_key: "fe1ba754dc47c4ce3830d00f6247da66"
server_key_iv: "eee2ec829ae867b50242bac51ccd3b1d"
server_key_version: "encryptionkey@0"
I20260811 04:39:03.659158 17691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "94601691df0d41bbad9fe4e1ad3ffc6c"
format_stamp: "Formatted at 2026-08-11 04:39:03 on dist-test-slave-0kls"
server_key: "fe1ba754dc47c4ce3830d00f6247da66"
server_key_iv: "eee2ec829ae867b50242bac51ccd3b1d"
server_key_version: "encryptionkey@0"
I20260811 04:39:03.662885 17691 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:03.665074 17707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:03.666325 17691 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:03.666458 17691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "94601691df0d41bbad9fe4e1ad3ffc6c"
format_stamp: "Formatted at 2026-08-11 04:39:03 on dist-test-slave-0kls"
server_key: "fe1ba754dc47c4ce3830d00f6247da66"
server_key_iv: "eee2ec829ae867b50242bac51ccd3b1d"
server_key_version: "encryptionkey@0"
I20260811 04:39:03.666553 17691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:03.687963 17691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:03.690896 17691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:03.691129 17691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:03.698205 17691 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:46097
I20260811 04:39:03.698228 17764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:46097 every 8 connection(s)
I20260811 04:39:03.699139 17691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:03.701897 17765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:03.703552 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17691
I20260811 04:39:03.703640 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:39:03.703925 3002 external_mini_cluster.cc:1428] Setting key d4318d7ef66deee4121afa25486df04c
I20260811 04:39:03.707705 17765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c: Bootstrap starting.
I20260811 04:39:03.709950 17765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:03.710624 17765 log.cc:824] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17678](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:03.712729 17765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c: No bootstrap required, opened a new log
I20260811 04:39:03.715873 17765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } }
I20260811 04:39:03.716131 17765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:03.716240 17765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94601691df0d41bbad9fe4e1ad3ffc6c, State: Initialized, Role: FOLLOWER
I20260811 04:39:03.716784 17765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } }
I20260811 04:39:03.716949 17765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:03.717046 17765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:03.717145 17765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:03.717563 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:03.705221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34490 (local address 127.2.238.190:46097)
0811 04:39:03.705617 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:03.705630 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:03.705664 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:39:03.706332 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:03.706349 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:03.706651 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:03.706966 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:03.707701 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:03.708719 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:03.709623 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:03.710200 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:03.710382 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:03.712931 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:03.712937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:03.712950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:03.712977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:03.714559 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:03.714990 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:03.714995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:03.715000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:03.715106 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:03.715387 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:03.715390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:03.715391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:03.715755 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:03.715994 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:03.716316 (+ 322us) server_negotiation.cc:299] Negotiation successful
0811 04:39:03.716679 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260811 04:39:03.718142 17765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } }
I20260811 04:39:03.718494 17765 leader_election.cc:302] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94601691df0d41bbad9fe4e1ad3ffc6c; no voters:
I20260811 04:39:03.718848 17765 leader_election.cc:288] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:39:03.718983 17770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:03.719251 17770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [term 1 LEADER]: Becoming Leader. State: Replica: 94601691df0d41bbad9fe4e1ad3ffc6c, State: Running, Role: LEADER
I20260811 04:39:03.719632 17770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } }
I20260811 04:39:03.720155 17765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:03.720829 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:39:03.721629 17772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } } }
I20260811 04:39:03.721812 17772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:03.721647 17771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94601691df0d41bbad9fe4e1ad3ffc6c. Latest consensus state: current_term: 1 leader_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94601691df0d41bbad9fe4e1ad3ffc6c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 46097 } } }
I20260811 04:39:03.722328 17771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:03.722394 17781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:03.725538 17781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:03.731033 17781 catalog_manager.cc:1347] Generated new cluster ID: 53c97cc3f17d4abdb61b1455660256ed
I20260811 04:39:03.731101 17781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:03.742794 17781 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:39:03.743669 17781 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:03.748994 17781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94601691df0d41bbad9fe4e1ad3ffc6c: Generated new TSK 0
I20260811 04:39:03.749641 17781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:39:03.777731 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17691
2026-08-11T04:39:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1425 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17792](info): set up 2 sockets
Aug 11 04:39:03 dist-test-slave-0kls krb5kdc[17792](info): commencing operation
krb5kdc: starting...
W20260811 04:39:04.812891 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:39:04 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:39:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:39:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:39:04.937156 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:05.051230 17805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:05.051553 17805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:05.051631 17805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:05.055352 17805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:05.055433 17805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:05.055470 17805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:05.055527 17805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:05.055567 17805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:05.060979 17805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:05.062139 17805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:05.063144 17805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:05.071532 17816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:05.071532 17813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.071646 17805 server_base.cc:1034] running on GCE node
W20260811 04:39:05.071517 17812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.072369 17805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:05.073433 17805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:05.074623 17805 hybrid_clock.cc:648] HybridClock initialized: now 1786423145074593 us; error 41 us; skew 500 ppm
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:05.087224 17805 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:05.088243 17805 webserver.cc:460] Webserver started at http://127.2.238.190:44619/ using document root <none> and password file <none>
I20260811 04:39:05.088868 17805 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:05.088936 17805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:05.089215 17805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:05.091499 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "769a46e9dcd348a68dacaa9b0edd1546"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "0a51bf114d9d0da6ea9a99b5a4805b34"
server_key_iv: "4df0560976a36ad8249d226cdbcbd1ee"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.092063 17805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "769a46e9dcd348a68dacaa9b0edd1546"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "0a51bf114d9d0da6ea9a99b5a4805b34"
server_key_iv: "4df0560976a36ad8249d226cdbcbd1ee"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.096553 17805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260811 04:39:05.099097 17823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.100440 17805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:39:05.100625 17805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "769a46e9dcd348a68dacaa9b0edd1546"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "0a51bf114d9d0da6ea9a99b5a4805b34"
server_key_iv: "4df0560976a36ad8249d226cdbcbd1ee"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.100754 17805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:05.114226 17805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:05.117179 17805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:05.117394 17805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:05.125231 17805 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39151
I20260811 04:39:05.125509 17877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39151 every 8 connection(s)
I20260811 04:39:05.126322 17805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:05.129057 17878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:05.134063 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17805
I20260811 04:39:05.134220 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:39:05.134519 3002 external_mini_cluster.cc:1428] Setting key 207b953b67b7278cc0b0b39f8eaa711e
I20260811 04:39:05.134706 17878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap starting.
I20260811 04:39:05.137212 17878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:05.138049 17878 log.cc:824] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:05.140280 17878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: No bootstrap required, opened a new log
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:05.144731 17878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:05.145054 17878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:05.145115 17878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Initialized, Role: FOLLOWER
I20260811 04:39:05.145622 17878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:05.145846 17878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:05.145949 17878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:05.146086 17878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:05.147035 17878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:05.147346 17878 leader_election.cc:302] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 769a46e9dcd348a68dacaa9b0edd1546; no voters:
I20260811 04:39:05.147667 17878 leader_election.cc:288] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:39:05.147732 17884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:05.147910 17884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 LEADER]: Becoming Leader. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Running, Role: LEADER
I20260811 04:39:05.148238 17884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:05.148962 17878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:05.149452 17884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:05.149623 17884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:05.149887 17885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 769a46e9dcd348a68dacaa9b0edd1546. Latest consensus state: current_term: 1 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:05.149973 17885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:05.150058 17889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:05.150372 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.136254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44650 (local address 127.2.238.190:39151)
0811 04:39:05.136708 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.136724 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.136759 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:39:05.137422 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.137439 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.137760 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.138102 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:05.139075 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.140062 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.141221 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.141643 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.141753 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.144576 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:05.144582 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:05.144597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:05.144624 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:05.146647 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.147151 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.147157 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.147163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.147279 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.147563 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.147567 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.147568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.148054 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:05.148283 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.149038 (+ 755us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.149335 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":147,"threads_started":1}
I20260811 04:39:05.154196 17889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:05.159559 17889 catalog_manager.cc:1347] Generated new cluster ID: 8f2ef170694c4462b0526db8c0ee86dd
I20260811 04:39:05.159626 17889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:05.178843 17889 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:39:05.180060 17889 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:05.186846 17889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Generated new TSK 0
I20260811 04:39:05.187561 17889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:39:05.225548 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:05.342974 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:05.343205 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:05.343243 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:05.346907 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:05.346959 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:05.347069 17905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:39:05.352103 17905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:05.353037 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:05.354023 17905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:05.363674 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:05.363669 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.363817 17905 server_base.cc:1034] running on GCE node
W20260811 04:39:05.363998 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.364411 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:05.365047 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:05.366207 17905 hybrid_clock.cc:648] HybridClock initialized: now 1786423145366188 us; error 30 us; skew 500 ppm
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:05.368872 17905 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:39:05.369858 17905 webserver.cc:460] Webserver started at http://127.2.238.129:35575/ using document root <none> and password file <none>
I20260811 04:39:05.370422 17905 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:05.370482 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:05.370743 17905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:05.372706 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "d5b276ea40644b3f92ee38c84668651b"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "1c03d4244447ae2750281b0acda50313"
server_key_iv: "dafb72446e8b6e6e7a498e95accd6f70"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.373256 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "d5b276ea40644b3f92ee38c84668651b"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "1c03d4244447ae2750281b0acda50313"
server_key_iv: "dafb72446e8b6e6e7a498e95accd6f70"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.376861 17905 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:39:05.379194 17920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.380390 17905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:39:05.380524 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "d5b276ea40644b3f92ee38c84668651b"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "1c03d4244447ae2750281b0acda50313"
server_key_iv: "dafb72446e8b6e6e7a498e95accd6f70"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.380648 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:05.393507 17905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:05.396202 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:05.396386 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:05.396937 17905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:05.397801 17905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:05.397873 17905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.397950 17905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:05.397993 17905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.406831 17905 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:34491
I20260811 04:39:05.406924 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:34491 every 8 connection(s)
I20260811 04:39:05.407753 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:39:05.413475 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 17905
I20260811 04:39:05.413635 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:39:05.414048 3002 external_mini_cluster.cc:1428] Setting key 3629fe0e6e6d840d7a023120e78f2939
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:05.421630 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.409571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:38621 (local address 127.2.238.190:39151)
0811 04:39:05.409732 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.409735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.410569 (+ 834us) server_negotiation.cc:407] Connection header received
0811 04:39:05.411800 (+ 1231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.411803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.411859 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.411963 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:05.413063 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.413622 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.415129 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.415411 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.415496 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.418086 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:05.418092 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:05.418100 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:05.418138 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:05.419507 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.420059 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.420063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.420065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.420156 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.420506 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.420510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.420512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.420702 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:05.420808 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.421333 (+ 525us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.421457 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:39:05.422490 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.409989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:38621)
0811 04:39:05.410427 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.410469 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.411598 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.412129 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.412136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.412484 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:05.412927 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.412939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.413820 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.413824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.415002 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.415024 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.415574 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.415578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.415731 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.416340 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:05.416355 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:05.417912 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:05.419646 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.419652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.419668 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.419946 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.420253 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.420257 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.420259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.420403 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.420822 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:05.420830 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:05.421092 (+ 262us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.421277 (+ 185us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.421603 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":128,"threads_started":1}
I20260811 04:39:05.423851 18039 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:05.424165 18039 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:39:05.424814 18039 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
I20260811 04:39:05.426486 17842 ts_manager.cc:194] Registered new tserver with Master: d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:39:05.427789 17842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:38621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:39:05.442848 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:05.557170 18046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:05.557404 18046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:05.557444 18046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:05.561312 18046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:05.561383 18046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:05.561440 18046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:39:05.566228 18046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:05.567164 18046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:05.568099 18046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:05.577924 18054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.577988 18046 server_base.cc:1034] running on GCE node
W20260811 04:39:05.577939 18051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:05.577939 18052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.578627 18046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:05.579283 18046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:05.580477 18046 hybrid_clock.cc:648] HybridClock initialized: now 1786423145580447 us; error 42 us; skew 500 ppm
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:05.583429 18046 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:39:05.584388 18046 webserver.cc:460] Webserver started at http://127.2.238.130:43051/ using document root <none> and password file <none>
I20260811 04:39:05.584988 18046 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:05.585052 18046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:05.585229 18046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:05.587384 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "23b190482d7917d7762fcb8c7b6a34e5"
server_key_iv: "4568debec1c2548aa68d77a20a092ab1"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.587914 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "23b190482d7917d7762fcb8c7b6a34e5"
server_key_iv: "4568debec1c2548aa68d77a20a092ab1"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.591799 18046 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:39:05.594328 18061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.595613 18046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:05.595741 18046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "23b190482d7917d7762fcb8c7b6a34e5"
server_key_iv: "4568debec1c2548aa68d77a20a092ab1"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.595855 18046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:05.618712 18046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:05.622170 18046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:05.622376 18046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:05.622994 18046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:05.623800 18046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:05.623845 18046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.623911 18046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:05.623941 18046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.632709 18046 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:42905
I20260811 04:39:05.632735 18180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:42905 every 8 connection(s)
I20260811 04:39:05.633621 18046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:39:05.639526 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18046
I20260811 04:39:05.639693 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:39:05.640017 3002 external_mini_cluster.cc:1428] Setting key 099bba6207533dfd5c05e1a651401ecf
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:05.647428 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.635447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:59675 (local address 127.2.238.190:39151)
0811 04:39:05.635631 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.635635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.636537 (+ 902us) server_negotiation.cc:407] Connection header received
0811 04:39:05.637719 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.637722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.637768 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.637886 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:05.638972 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.639704 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.640919 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.641178 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.641295 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.643967 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:05.643970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:05.643971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:05.643991 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:05.645263 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.645866 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.645868 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.645870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.645957 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.646372 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.646373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.646374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.646521 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:05.646589 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.647232 (+ 643us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.647318 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":63}
I20260811 04:39:05.648542 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.635907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:59675)
0811 04:39:05.636371 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.636426 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.637503 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.638016 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.638022 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.638358 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:05.638841 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.638854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.639850 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.639854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.640784 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.640795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.641338 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.641342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.641506 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.642144 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:05.642160 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:05.643797 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:05.645371 (+ 1574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.645379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.645394 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.645738 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.646051 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.646057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.646061 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.646282 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.646637 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:05.646697 (+ 60us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:05.646997 (+ 300us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.647201 (+ 204us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.647528 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":142,"threads_started":1}
I20260811 04:39:05.649896 18181 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:05.650163 18181 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:05.650697 18181 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:39:05.651732 17842 ts_manager.cc:194] Registered new tserver with Master: 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905)
I20260811 04:39:05.652240 17842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:59675
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:39:05.668689 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:05.778599 18186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:05.778854 18186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:05.778898 18186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:05.782827 18186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:05.782882 18186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:05.782939 18186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:39:05.788143 18186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:05.789179 18186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:05.790285 18186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:05.799533 18194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:05.799571 18191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:05.799561 18192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:05.799841 18186 server_base.cc:1034] running on GCE node
I20260811 04:39:05.800391 18186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:05.801020 18186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:05.802218 18186 hybrid_clock.cc:648] HybridClock initialized: now 1786423145802179 us; error 49 us; skew 500 ppm
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:05.805011 18186 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:39:05.806020 18186 webserver.cc:460] Webserver started at http://127.2.238.131:45665/ using document root <none> and password file <none>
I20260811 04:39:05.806589 18186 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:05.806682 18186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:05.806937 18186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:05.808983 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "6ee2632780e0eec287d9a2185e88ab2a"
server_key_iv: "4c2f65c031784cc4248e2d42aa562cb7"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.809568 18186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "6ee2632780e0eec287d9a2185e88ab2a"
server_key_iv: "4c2f65c031784cc4248e2d42aa562cb7"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.814615 18186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260811 04:39:05.818081 18202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.819744 18186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:05.819885 18186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "6ee2632780e0eec287d9a2185e88ab2a"
server_key_iv: "4c2f65c031784cc4248e2d42aa562cb7"
server_key_version: "encryptionkey@0"
I20260811 04:39:05.820022 18186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:05.843533 18186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:05.846549 18186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:05.846786 18186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:05.847414 18186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:05.848342 18186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:05.848392 18186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.848459 18186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:05.848492 18186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:05.856943 18186 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:33853
I20260811 04:39:05.857064 18321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:33853 every 8 connection(s)
I20260811 04:39:05.857944 18186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:39:05.865899 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18186
I20260811 04:39:05.866077 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:39:05.866403 3002 external_mini_cluster.cc:1428] Setting key 44c8490daacac4e8adf3883274a28100
Aug 11 04:39:05 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423145, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:05.871412 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.859776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40589 (local address 127.2.238.190:39151)
0811 04:39:05.859948 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.859953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.860788 (+ 835us) server_negotiation.cc:407] Connection header received
0811 04:39:05.862090 (+ 1302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.862092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.862142 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.862217 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:05.863336 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.863853 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.864942 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.865156 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.865239 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.867857 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:05.867861 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:05.867863 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:05.867895 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:05.869341 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.869881 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.869886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.869888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.869987 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.870338 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.870341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.870343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.870526 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:05.870635 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.871126 (+ 491us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.871245 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260811 04:39:05.872157 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.860193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:40589)
0811 04:39:05.860639 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.860681 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.861898 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.862374 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.862381 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.862715 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:05.863200 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.863214 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.863961 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.863963 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.864822 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.864831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.865285 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.865288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.865421 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.866104 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:05.866141 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:05.867696 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:05.869465 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.869470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.869485 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.869745 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.870089 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:05.870093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:05.870096 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:05.870237 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:05.870635 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:05.870642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:05.870877 (+ 235us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.871056 (+ 179us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.871367 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":126,"threads_started":1}
I20260811 04:39:05.873210 18322 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:05.873415 18322 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:05.873906 18322 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
I20260811 04:39:05.874825 17842 ts_manager.cc:194] Registered new tserver with Master: 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
I20260811 04:39:05.875397 17842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:40589
I20260811 04:39:05.880193 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:39:05.889325 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.882201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44664 (local address 127.2.238.190:39151)
0811 04:39:05.882317 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.882321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.882483 (+ 162us) server_negotiation.cc:407] Connection header received
0811 04:39:05.882571 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.882573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.882617 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.882684 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:05.883356 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.883852 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.884745 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.884949 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.885028 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.886511 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:05.886514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:05.886516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:05.886537 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:05.887737 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.888101 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.888103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.888104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.888176 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:05.888402 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:05.888404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:05.888405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:05.888603 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:05.888714 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.889094 (+ 380us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.889204 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:39:05.892534 17842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:39:05.895730 17842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:39:05.910604 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.904821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:42905 (local address 127.0.0.1:38850)
0811 04:39:05.905404 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.905418 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.905497 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.906405 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.906409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.906427 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:05.906516 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.906521 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.907916 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.907922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.909141 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.909153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.910054 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.910175 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910213 (+ 38us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.910331 (+ 118us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.910446 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":108,"threads_started":1}
I20260811 04:39:05.910805 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.904865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.2.238.131:33853)
0811 04:39:05.905876 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.905889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.905911 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:39:05.905984 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.905991 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.906218 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.906728 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:05.906885 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.907977 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.909313 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.910166 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910306 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910420 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.910550 (+ 130us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.910665 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":131,"threads_started":1}
I20260811 04:39:05.910846 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.904226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:33853 (local address 127.0.0.1:32886)
0811 04:39:05.905064 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.905079 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.905173 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.906454 (+ 1281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.906457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.906476 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:05.906569 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.906578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.908083 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.908086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.909182 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.909192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.910323 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.910417 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910437 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.910504 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.910557 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":102,"threads_started":1}
I20260811 04:39:05.911118 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.904010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34491 (local address 127.0.0.1:42742)
0811 04:39:05.904358 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:05.904405 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:05.904524 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:05.905313 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:05.905320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:05.905344 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:05.905497 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.905503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.906915 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.906921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.907779 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:05.907788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.910484 (+ 2696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:05.910583 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910605 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 04:39:05.910679 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 04:39:05.910951 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":109,"threads_started":1}
I20260811 04:39:05.911198 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.905271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.2.238.130:42905)
0811 04:39:05.905608 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.905620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.905645 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:39:05.905722 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.905727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.906284 (+ 557us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.906434 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:05.906620 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.907753 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.909257 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.909933 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910054 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910204 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.910969 (+ 765us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.911072 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":155,"threads_started":1}
I20260811 04:39:05.911242 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:05.904180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.2.238.129:34491)
0811 04:39:05.904578 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:05.904594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:05.904632 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:39:05.904715 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:05.904721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:05.904925 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:05.905106 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:05.905647 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.906799 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.909299 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:05.910166 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:05.910284 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:05.910420 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:05.911020 (+ 600us) server_negotiation.cc:299] Negotiation successful
0811 04:39:05.911106 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":81,"threads_started":1}
I20260811 04:39:05.914172 17972 tablet_service.cc:1467] Processing CreateTablet for tablet 71766d5583884f2faf99615f66ef0430 (DEFAULT_TABLE table=test-table [id=95f9da6bd57141f981bd158a7a809c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:05.915212 17972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71766d5583884f2faf99615f66ef0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:05.918124 18254 tablet_service.cc:1467] Processing CreateTablet for tablet 71766d5583884f2faf99615f66ef0430 (DEFAULT_TABLE table=test-table [id=95f9da6bd57141f981bd158a7a809c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:05.919044 18254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71766d5583884f2faf99615f66ef0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:05.920622 18113 tablet_service.cc:1467] Processing CreateTablet for tablet 71766d5583884f2faf99615f66ef0430 (DEFAULT_TABLE table=test-table [id=95f9da6bd57141f981bd158a7a809c7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:05.921667 18113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71766d5583884f2faf99615f66ef0430. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:05.922399 18339 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Bootstrap starting.
I20260811 04:39:05.924996 18339 tablet_bootstrap.cc:654] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:05.925908 18340 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Bootstrap starting.
I20260811 04:39:05.925936 18339 log.cc:824] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:05.928146 18339 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: No bootstrap required, opened a new log
I20260811 04:39:05.928169 18340 tablet_bootstrap.cc:654] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:05.928370 18339 ts_tablet_manager.cc:1397] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:05.928824 18342 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Bootstrap starting.
I20260811 04:39:05.929451 18340 log.cc:824] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:05.931147 18340 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: No bootstrap required, opened a new log
I20260811 04:39:05.931345 18340 ts_tablet_manager.cc:1397] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:39:05.931504 18342 tablet_bootstrap.cc:654] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:05.932374 18342 log.cc:824] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:05.933018 18339 raft_consensus.cc:348] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.933290 18339 raft_consensus.cc:374] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:05.933414 18339 raft_consensus.cc:729] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b276ea40644b3f92ee38c84668651b, State: Initialized, Role: FOLLOWER
I20260811 04:39:05.933954 18339 consensus_queue.cc:260] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.934430 18342 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: No bootstrap required, opened a new log
I20260811 04:39:05.934651 18342 ts_tablet_manager.cc:1397] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:05.934789 18039 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
I20260811 04:39:05.934991 18340 raft_consensus.cc:348] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.935220 18340 raft_consensus.cc:374] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:05.935281 18340 raft_consensus.cc:729] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4109df9aa84f8ba59f625dbfe8d686, State: Initialized, Role: FOLLOWER
I20260811 04:39:05.935683 18340 consensus_queue.cc:260] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.936190 18339 ts_tablet_manager.cc:1428] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260811 04:39:05.936743 18340 ts_tablet_manager.cc:1428] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260811 04:39:05.936807 18322 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
I20260811 04:39:05.938119 18342 raft_consensus.cc:348] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.938387 18342 raft_consensus.cc:374] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:05.938457 18342 raft_consensus.cc:729] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd85a26d1564aa2816ee7b790cfcebf, State: Initialized, Role: FOLLOWER
I20260811 04:39:05.938898 18342 consensus_queue.cc:260] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:05.939667 18181 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
I20260811 04:39:05.940804 18342 ts_tablet_manager.cc:1428] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:06.098624 18345 raft_consensus.cc:482] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:06.098877 18345 raft_consensus.cc:504] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:06.100153 18345 leader_election.cc:288] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
I20260811 04:39:06.103950 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.100484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:49479 (local address 127.2.238.130:42905)
0811 04:39:06.100654 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.100659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.100935 (+ 276us) server_negotiation.cc:407] Connection header received
0811 04:39:06.101112 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.101120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.101161 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.101235 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:06.101523 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.102028 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103039 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.103380 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103503 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.103553 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.103785 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.103849 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:39:06.103950 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.100524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:60783 (local address 127.2.238.131:33853)
0811 04:39:06.100688 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.100693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.100877 (+ 184us) server_negotiation.cc:407] Connection header received
0811 04:39:06.101065 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.101067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.101109 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.101185 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:06.101483 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.101993 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103032 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.103392 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103508 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.103559 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.103787 (+ 228us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.103849 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 04:39:06.103976 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.100438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:42905 (local address 127.2.238.129:49479)
0811 04:39:06.100825 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.100837 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.100985 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.101257 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.101259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.101306 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:06.101413 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.101417 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.102132 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.102136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.102917 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.102926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.103503 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103506 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.103645 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.103676 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.103758 (+ 82us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.103822 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":87,"spinlock_wait_cycles":768,"thread_start_us":129,"threads_started":1}
I20260811 04:39:06.103982 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.100413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:33853 (local address 127.2.238.129:60783)
0811 04:39:06.100776 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.100789 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.100919 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.101207 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.101210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.101224 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:06.101379 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.101384 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.102101 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.102104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.102913 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.102922 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.103507 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.103510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.103636 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.103692 (+ 56us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.103771 (+ 79us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.103834 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":76,"spinlock_wait_cycles":32128,"thread_start_us":104,"threads_started":1}
I20260811 04:39:06.104619 18134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71766d5583884f2faf99615f66ef0430" candidate_uuid: "d5b276ea40644b3f92ee38c84668651b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" is_pre_election: true
I20260811 04:39:06.104619 18275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71766d5583884f2faf99615f66ef0430" candidate_uuid: "d5b276ea40644b3f92ee38c84668651b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" is_pre_election: true
I20260811 04:39:06.104898 18275 raft_consensus.cc:2454] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b276ea40644b3f92ee38c84668651b in term 0.
I20260811 04:39:06.104897 18134 raft_consensus.cc:2454] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b276ea40644b3f92ee38c84668651b in term 0.
I20260811 04:39:06.105389 17923 leader_election.cc:302] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686, d5b276ea40644b3f92ee38c84668651b; no voters:
I20260811 04:39:06.105710 18345 raft_consensus.cc:2781] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:39:06.105849 18345 raft_consensus.cc:482] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:39:06.105966 18345 raft_consensus.cc:3037] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:06.107039 18345 raft_consensus.cc:504] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:06.107388 18345 leader_election.cc:288] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 1 election: Requested vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
I20260811 04:39:06.107728 18275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71766d5583884f2faf99615f66ef0430" candidate_uuid: "d5b276ea40644b3f92ee38c84668651b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
I20260811 04:39:06.107810 18134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71766d5583884f2faf99615f66ef0430" candidate_uuid: "d5b276ea40644b3f92ee38c84668651b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
I20260811 04:39:06.107828 18275 raft_consensus.cc:3037] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:06.107926 18134 raft_consensus.cc:3037] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:06.108896 18275 raft_consensus.cc:2454] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b276ea40644b3f92ee38c84668651b in term 1.
W20260811 04:39:06.108909 18323 tablet.cc:2369] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:06.109238 18134 raft_consensus.cc:2454] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b276ea40644b3f92ee38c84668651b in term 1.
I20260811 04:39:06.109570 17923 leader_election.cc:302] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd85a26d1564aa2816ee7b790cfcebf, d5b276ea40644b3f92ee38c84668651b; no voters:
I20260811 04:39:06.109742 18345 raft_consensus.cc:2781] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:06.110090 18345 raft_consensus.cc:686] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 LEADER]: Becoming Leader. State: Replica: d5b276ea40644b3f92ee38c84668651b, State: Running, Role: LEADER
I20260811 04:39:06.110435 18345 consensus_queue.cc:237] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:06.114038 17842 catalog_manager.cc:5495] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b276ea40644b3f92ee38c84668651b (127.2.238.129). New cstate: current_term: 1 leader_uuid: "d5b276ea40644b3f92ee38c84668651b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } health_report { overall_health: UNKNOWN } } }
W20260811 04:39:06.134824 18182 tablet.cc:2369] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:06.144644 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.141478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42744 (local address 127.2.238.129:34491)
0811 04:39:06.141612 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.141615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.141628 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:39:06.141760 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.141763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.141870 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.141953 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:06.142197 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.142641 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.143407 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.143593 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.143682 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.143945 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:06.144302 (+ 357us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:39:06.144387 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.144441 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.144484 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:39:06.150269 18134 raft_consensus.cc:1261] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Refusing update from remote peer d5b276ea40644b3f92ee38c84668651b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:06.150251 18275 raft_consensus.cc:1261] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Refusing update from remote peer d5b276ea40644b3f92ee38c84668651b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:06.151000 18350 consensus_queue.cc:1035] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:06.151187 18345 consensus_queue.cc:1035] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:06.157680 18354 mvcc.cc:205] Tried to move back new op lower bound from 7317189206623764480 to 7317189206471417856. Current Snapshot: MvccSnapshot[applied={T|T < 7317189206623764480}]
I20260811 04:39:06.160290 18356 mvcc.cc:205] Tried to move back new op lower bound from 7317189206623764480 to 7317189206471417856. Current Snapshot: MvccSnapshot[applied={T|T < 7317189206623764480 or (T in {7317189206623764480})}]
I20260811 04:39:06.161777 18355 mvcc.cc:205] Tried to move back new op lower bound from 7317189206623764480 to 7317189206471417856. Current Snapshot: MvccSnapshot[applied={T|T < 7317189206623764480 or (T in {7317189206623764480})}]
I20260811 04:39:06.166771 17842 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:06.166920 17842 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:06.169344 17842 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Sending DeleteTablet for 3 replicas of tablet 71766d5583884f2faf99615f66ef0430
I20260811 04:39:06.170161 18113 tablet_service.cc:1514] Processing DeleteTablet for tablet 71766d5583884f2faf99615f66ef0430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:06 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:38850
I20260811 04:39:06.170163 18254 tablet_service.cc:1514] Processing DeleteTablet for tablet 71766d5583884f2faf99615f66ef0430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:06 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:32886
I20260811 04:39:06.170260 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17905
I20260811 04:39:06.172405 18364 tablet_replica.cc:330] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: stopping tablet replica
I20260811 04:39:06.172698 18364 raft_consensus.cc:2229] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:06.173007 18364 raft_consensus.cc:2258] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:06.173382 18365 tablet_replica.cc:330] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: stopping tablet replica
I20260811 04:39:06.174048 18365 raft_consensus.cc:2229] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:06.174336 18365 raft_consensus.cc:2258] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:06.174815 18364 ts_tablet_manager.cc:1905] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:06.175608 18365 ts_tablet_manager.cc:1905] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:06.177577 18364 ts_tablet_manager.cc:1918] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:06.177691 18364 log.cc:1196] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/71766d5583884f2faf99615f66ef0430
I20260811 04:39:06.177768 18365 ts_tablet_manager.cc:1918] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:06.177924 18365 log.cc:1196] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/71766d5583884f2faf99615f66ef0430
I20260811 04:39:06.178028 18364 ts_tablet_manager.cc:1939] T 71766d5583884f2faf99615f66ef0430 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting consensus metadata
I20260811 04:39:06.178249 18365 ts_tablet_manager.cc:1939] T 71766d5583884f2faf99615f66ef0430 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting consensus metadata
I20260811 04:39:06.179270 17828 catalog_manager.cc:4841] TS 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): tablet 71766d5583884f2faf99615f66ef0430 (table test-table [id=95f9da6bd57141f981bd158a7a809c7d]) successfully deleted
I20260811 04:39:06.179328 17825 catalog_manager.cc:4841] TS 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): tablet 71766d5583884f2faf99615f66ef0430 (table test-table [id=95f9da6bd57141f981bd158a7a809c7d]) successfully deleted
W20260811 04:39:06.180478 17828 connection.cc:535] client connection to 127.2.238.129:34491 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260811 04:39:06.180631 17828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 04:39:06.181427 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18046
I20260811 04:39:06.181664 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.181315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34491 (local address 127.0.0.1:42746)
0811 04:39:06.181460 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.181551 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 04:39:06.184718 17828 catalog_manager.cc:4584] TS d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 71766d5583884f2faf99615f66ef0430: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34491: connect: Connection refused (error 111)
I20260811 04:39:06.189316 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18186
I20260811 04:39:06.196252 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 17805
I20260811 04:39:06.204109 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--webserver_interface=127.2.238.190
--webserver_port=44619
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:06.317739 18367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:06.318166 18367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:06.318257 18367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:06.322499 18367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:06.322592 18367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:06.322633 18367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:06.322692 18367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:06.322729 18367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:06.328112 18367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:06.329216 18367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:06.330327 18367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:06.344363 18375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.344352 18377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.344349 18373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.344636 18367 server_base.cc:1034] running on GCE node
I20260811 04:39:06.345253 18367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:06.346371 18367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:06.347558 18367 hybrid_clock.cc:648] HybridClock initialized: now 1786423146347534 us; error 33 us; skew 500 ppm
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:06.350425 18367 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:06.351289 18367 webserver.cc:460] Webserver started at http://127.2.238.190:44619/ using document root <none> and password file <none>
I20260811 04:39:06.351836 18367 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:06.351909 18367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:06.355705 18367 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 04:39:06.357719 18384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.358937 18367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:06.359066 18367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "769a46e9dcd348a68dacaa9b0edd1546"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "0a51bf114d9d0da6ea9a99b5a4805b34"
server_key_iv: "4df0560976a36ad8249d226cdbcbd1ee"
server_key_version: "encryptionkey@0"
I20260811 04:39:06.359637 18367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:06.377516 18367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:06.380991 18367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:06.381217 18367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:06.388904 18367 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39151
I20260811 04:39:06.389221 18438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39151 every 8 connection(s)
I20260811 04:39:06.390139 18367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:06.390589 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18367
I20260811 04:39:06.391106 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:34491
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=35575
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:06.394801 18439 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:39:06.396579 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap starting.
I20260811 04:39:06.414629 18439 log.cc:824] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:06.423427 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:06.423821 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap complete.
I20260811 04:39:06.428129 18439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:06.428525 18439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Initialized, Role: FOLLOWER
I20260811 04:39:06.429105 18439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:06.429235 18439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:06.429334 18439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:06.429455 18439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:39:06.431546 18439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:06.431915 18439 leader_election.cc:302] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 769a46e9dcd348a68dacaa9b0edd1546; no voters:
I20260811 04:39:06.432262 18439 leader_election.cc:288] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 04:39:06.432389 18443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 04:39:06.432655 18443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 LEADER]: Becoming Leader. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Running, Role: LEADER
I20260811 04:39:06.433034 18443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:06.433629 18439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:06.434655 18445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 769a46e9dcd348a68dacaa9b0edd1546. Latest consensus state: current_term: 2 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:06.434803 18445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:06.435161 18452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:06.435174 18444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:06.435302 18444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:06.440483 18452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:06.441140 18452 catalog_manager.cc:1259] Loaded cluster ID: 8f2ef170694c4462b0526db8c0ee86dd
I20260811 04:39:06.441198 18452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:06.442673 18452 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260811 04:39:06.447130 18452 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260811 04:39:06.452606 18452 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:06.453346 18452 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Loaded TSK: 0
I20260811 04:39:06.454598 18452 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:39:06.513635 18441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:06.513927 18441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:06.513998 18441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:06.517725 18441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:06.517921 18441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:06.518042 18441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:39:06.523142 18441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:34491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=35575
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:06.524369 18441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:06.525282 18441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:06.535666 18468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.536077 18441 server_base.cc:1034] running on GCE node
W20260811 04:39:06.535642 18467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.535686 18470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.537038 18441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:06.537858 18441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:06.541144 18441 hybrid_clock.cc:648] HybridClock initialized: now 1786423146541106 us; error 52 us; skew 500 ppm
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:06.544150 18441 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:39:06.545015 18441 webserver.cc:460] Webserver started at http://127.2.238.129:35575/ using document root <none> and password file <none>
I20260811 04:39:06.545568 18441 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:06.545661 18441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:06.550100 18441 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260811 04:39:06.552309 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.553609 18441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:06.553745 18441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "d5b276ea40644b3f92ee38c84668651b"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "1c03d4244447ae2750281b0acda50313"
server_key_iv: "dafb72446e8b6e6e7a498e95accd6f70"
server_key_version: "encryptionkey@0"
I20260811 04:39:06.554314 18441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:06.566807 18441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:06.569847 18441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:06.570047 18441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:06.570643 18441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:06.571755 18484 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 04:39:06.575680 18441 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 04:39:06.575740 18441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260811 04:39:06.575804 18441 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 04:39:06.577412 18441 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 04:39:06.577466 18441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 04:39:06.577728 18484 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Bootstrap starting.
I20260811 04:39:06.588454 18441 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:34491
I20260811 04:39:06.588488 18599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:34491 every 8 connection(s)
I20260811 04:39:06.589702 18441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:39:06.597671 18484 log.cc:824] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:06.599177 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18441
I20260811 04:39:06.599702 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:42905
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=43051
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:06.606935 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.591876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:37465)
0811 04:39:06.592477 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.592525 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.593375 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.594531 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.594538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.594944 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:06.595390 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.595404 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.596656 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.596659 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.597460 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.597470 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.598367 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.598370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.598504 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.599230 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:06.599279 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:06.600966 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:06.603713 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.603718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.603737 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.603988 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.604457 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.604460 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.604462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.604622 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.605251 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:06.605257 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:06.605508 (+ 251us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.605719 (+ 211us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.606078 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":191,"threads_started":1}
I20260811 04:39:06.606922 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.591864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:37465 (local address 127.2.238.190:39151)
0811 04:39:06.592211 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.592232 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.592676 (+ 444us) server_negotiation.cc:407] Connection header received
0811 04:39:06.593808 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.593818 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.594148 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.594471 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:06.595580 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.596476 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.597599 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.598217 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.598398 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.601188 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:06.601194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:06.601206 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:06.601236 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:06.603578 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.604159 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.604165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.604169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.604313 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.604731 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.604737 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.604739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.605111 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:06.605265 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.605869 (+ 604us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.606131 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":116,"threads_started":1}
I20260811 04:39:06.607671 18484 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:06.608161 18484 tablet_bootstrap.cc:492] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Bootstrap complete.
I20260811 04:39:06.609472 18600 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:06.609720 18484 ts_tablet_manager.cc:1397] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.018s
I20260811 04:39:06.609745 18600 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:06.610859 18600 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
I20260811 04:39:06.613224 18398 ts_manager.cc:194] Registered new tserver with Master: d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491)
I20260811 04:39:06.613392 18484 raft_consensus.cc:348] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:06.613900 18484 raft_consensus.cc:729] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b276ea40644b3f92ee38c84668651b, State: Initialized, Role: FOLLOWER
I20260811 04:39:06.614396 18484 consensus_queue.cc:260] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } } peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } }
I20260811 04:39:06.615350 18484 ts_tablet_manager.cc:1428] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:06.616961 18398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:37465
I20260811 04:39:06.619663 18600 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:39:06.627065 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.616475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.2.238.129:34491)
0811 04:39:06.617138 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.617151 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.617172 (+ 21us) server_negotiation.cc:407] Connection header received
0811 04:39:06.617233 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.617238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.617433 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.617565 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:06.618384 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.619234 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.620329 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.620945 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.621059 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.623574 (+ 2515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:06.623583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:06.623592 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:06.623612 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:06.625095 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.625675 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.625683 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.625685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.625771 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.626119 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.626121 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.626122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.626335 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:06.626435 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.626820 (+ 385us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.626953 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":569,"thread_start_us":102,"threads_started":1}
I20260811 04:39:06.627066 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.616361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34491 (local address 127.0.0.1:42750)
0811 04:39:06.616805 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.616856 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.616984 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.617604 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.617610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.618132 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:06.618254 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.618261 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.619366 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.619372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.620193 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.620205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.621174 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.621178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.621286 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.621945 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:06.621959 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:06.623439 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:06.625200 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.625209 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.625214 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.625529 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.625927 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.625930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.625932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.626026 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.626455 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:06.626462 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:06.626593 (+ 131us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.626737 (+ 144us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.626909 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260811 04:39:06.628047 18530 tablet_service.cc:1514] Processing DeleteTablet for tablet 71766d5583884f2faf99615f66ef0430 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:06 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:42750
I20260811 04:39:06.628865 18609 tablet_replica.cc:330] stopping tablet replica
I20260811 04:39:06.629184 18609 raft_consensus.cc:2229] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:06.629434 18609 raft_consensus.cc:2258] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:06.630429 18609 ts_tablet_manager.cc:1905] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:06.634059 18609 ts_tablet_manager.cc:1918] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:06.634138 18609 log.cc:1196] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/71766d5583884f2faf99615f66ef0430
I20260811 04:39:06.634588 18609 ts_tablet_manager.cc:1939] T 71766d5583884f2faf99615f66ef0430 P d5b276ea40644b3f92ee38c84668651b: Deleting consensus metadata
I20260811 04:39:06.636504 18387 catalog_manager.cc:4841] TS d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491): tablet 71766d5583884f2faf99615f66ef0430 (table test-table [id=95f9da6bd57141f981bd158a7a809c7d]) successfully deleted
W20260811 04:39:06.716980 18605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:06.717216 18605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:06.717254 18605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:06.721092 18605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:06.721136 18605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:06.721187 18605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:39:06.725772 18605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=43051
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:06.726725 18605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:06.727670 18605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:06.737346 18618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.737339 18614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.737690 18605 server_base.cc:1034] running on GCE node
W20260811 04:39:06.737695 18615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.738188 18605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:06.738838 18605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:06.740028 18605 hybrid_clock.cc:648] HybridClock initialized: now 1786423146740009 us; error 29 us; skew 500 ppm
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:06.742908 18605 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:39:06.743709 18605 webserver.cc:460] Webserver started at http://127.2.238.130:43051/ using document root <none> and password file <none>
I20260811 04:39:06.744227 18605 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:06.744341 18605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:06.748214 18605 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:06.750370 18625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.751518 18605 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:06.751657 18605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "23b190482d7917d7762fcb8c7b6a34e5"
server_key_iv: "4568debec1c2548aa68d77a20a092ab1"
server_key_version: "encryptionkey@0"
I20260811 04:39:06.752200 18605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:06.787386 18605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:06.790755 18605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:06.790977 18605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:06.791601 18605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:06.792491 18605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:06.792598 18605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.792728 18605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:06.792789 18605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.801527 18605 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:42905
I20260811 04:39:06.801568 18748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:42905 every 8 connection(s)
I20260811 04:39:06.802541 18605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:39:06.808763 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18605
I20260811 04:39:06.809182 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:33853
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=45665
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:06.816915 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.804302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:49071 (local address 127.2.238.190:39151)
0811 04:39:06.804434 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.804437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.805191 (+ 754us) server_negotiation.cc:407] Connection header received
0811 04:39:06.806609 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.806613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.806682 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.806773 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:06.807969 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.808550 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.809940 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.810326 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.810449 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.813000 (+ 2551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:06.813006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:06.813012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:06.813045 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:06.814751 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.815262 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.815269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.815273 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.815369 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.815698 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.815704 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.815708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.815970 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:06.816090 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.816513 (+ 423us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.816678 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260811 04:39:06.817489 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.804638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:49071)
0811 04:39:06.805047 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.805087 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.806351 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.806950 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.806957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.807326 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:06.807791 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.807805 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.808710 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.808713 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.809760 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.809771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.810477 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.810480 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.810622 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.811217 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:06.811233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:06.812831 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:06.814889 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.814893 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.814903 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.815159 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.815476 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.815478 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.815479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.815605 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.816082 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:06.816087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:06.816287 (+ 200us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.816425 (+ 138us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.816655 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":103,"threads_started":1}
I20260811 04:39:06.818785 18749 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:06.819034 18749 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:06.819511 18749 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
I20260811 04:39:06.820628 18398 ts_manager.cc:194] Registered new tserver with Master: 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905)
I20260811 04:39:06.821187 18398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:49071
W20260811 04:39:06.921391 18752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:06.921639 18752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:06.921681 18752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:06.925590 18752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:06.925653 18752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:06.925740 18752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:39:06.930763 18752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=45665
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:06.931710 18752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:06.932693 18752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:06.942282 18758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.942314 18759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:06.942431 18761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:06.942571 18752 server_base.cc:1034] running on GCE node
I20260811 04:39:06.942937 18752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:06.943531 18752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:06.944725 18752 hybrid_clock.cc:648] HybridClock initialized: now 1786423146944708 us; error 35 us; skew 500 ppm
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:06.947319 18752 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:39:06.948165 18752 webserver.cc:460] Webserver started at http://127.2.238.131:45665/ using document root <none> and password file <none>
I20260811 04:39:06.948716 18752 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:06.948820 18752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:06.952664 18752 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:06.954733 18770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.956063 18752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:06.956220 18752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "6ee2632780e0eec287d9a2185e88ab2a"
server_key_iv: "4c2f65c031784cc4248e2d42aa562cb7"
server_key_version: "encryptionkey@0"
I20260811 04:39:06.956727 18752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:06.968568 18752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:06.971503 18752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:06.971647 18752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:06.972182 18752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:06.972920 18752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:06.972966 18752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.973006 18752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:06.973021 18752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:06.982096 18752 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:33853
I20260811 04:39:06.982120 18890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:33853 every 8 connection(s)
I20260811 04:39:06.983054 18752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:39:06.986986 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18752
Aug 11 04:39:06 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423146, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:06.995950 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.987672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44676 (local address 127.2.238.190:39151)
0811 04:39:06.987919 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.987926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.988159 (+ 233us) server_negotiation.cc:407] Connection header received
0811 04:39:06.988311 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.988314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.988363 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.988456 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:06.989221 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.989749 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.990649 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.990900 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.991029 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.992633 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:06.992636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:06.992638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:06.992661 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:06.994349 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.994769 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.994773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.994775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.994858 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.995132 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.995134 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.995135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.995300 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:06.995478 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.995672 (+ 194us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.995773 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":164,"thread_start_us":72,"threads_started":1}
I20260811 04:39:06.997115 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.984868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53619 (local address 127.2.238.190:39151)
0811 04:39:06.985001 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:06.985005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:06.986353 (+ 1348us) server_negotiation.cc:407] Connection header received
0811 04:39:06.987319 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:06.987322 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:06.987384 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:06.987482 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:06.988721 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.989269 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.990465 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.990694 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.990814 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.993426 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:06.993431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:06.993437 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:06.993484 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:06.994949 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.995468 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.995471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.995472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.995548 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:06.995984 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:06.995987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:06.995989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:06.996214 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:06.996328 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:06.996855 (+ 527us) server_negotiation.cc:299] Negotiation successful
0811 04:39:06.996952 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260811 04:39:06.998064 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:06.985260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:53619)
0811 04:39:06.986177 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:06.986222 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:06.987095 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:06.987657 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:06.987665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:06.988080 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:06.988545 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.988559 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.989403 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.989407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.990334 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:06.990344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:06.990867 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:06.990871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:06.991054 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:06.991575 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:06.991591 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:06.993268 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:06.995037 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.995042 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.995056 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.995360 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.995673 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:06.995676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:06.995677 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:06.995845 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:06.996350 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:06.996356 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:06.996632 (+ 276us) client_negotiation.cc:769] Sending connection context
0811 04:39:06.996784 (+ 152us) client_negotiation.cc:240] Negotiation successful
0811 04:39:06.997107 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":596,"threads_started":1}
I20260811 04:39:06.999121 18891 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39151
I20260811 04:39:06.999365 18891 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:06.999866 18891 heartbeater.cc:511] Master 127.2.238.190:39151 requested a full tablet report, sending...
I20260811 04:39:07.000813 18398 ts_manager.cc:194] Registered new tserver with Master: 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
I20260811 04:39:07.001398 18398 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:53619
I20260811 04:39:07.009382 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:39:07.013568 18602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.009961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44692 (local address 127.2.238.190:39151)
0811 04:39:07.010097 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.010101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.010278 (+ 177us) server_negotiation.cc:407] Connection header received
0811 04:39:07.010468 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.010470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.010554 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.010662 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:07.010895 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.011369 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.012239 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.012456 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.012575 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.012885 (+ 310us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:07.013211 (+ 326us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:39:07.013337 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.013401 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.013446 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:39:07.014199 18398 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:39:07.017309 18398 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:39:07.025115 18530 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 (DEFAULT_TABLE table=test-table [id=f929e7b861fc499491a1e7806fb3c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:07.025801 18530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:07.028680 18484 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Bootstrap starting.
I20260811 04:39:07.029451 18607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.023981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:42905 (local address 127.0.0.1:38862)
0811 04:39:07.024105 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.024119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:07.024191 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:07.025015 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:07.025018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:07.025038 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:07.025127 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.025132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.026602 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.026608 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.027852 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.027864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.028996 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.029002 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.029111 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.029173 (+ 62us) client_negotiation.cc:769] Sending connection context
0811 04:39:07.029234 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 04:39:07.029323 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 04:39:07.029642 18484 tablet_bootstrap.cc:654] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:07.029700 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.024410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:33853 (local address 127.0.0.1:32892)
0811 04:39:07.024646 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.024661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:07.024749 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:07.025544 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:07.025547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:07.025565 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:07.025646 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.025652 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.027255 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.027259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.028237 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.028245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.029262 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.029266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.029372 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.029398 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:39:07.029451 (+ 53us) client_negotiation.cc:240] Negotiation successful
0811 04:39:07.029529 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260811 04:39:07.030370 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.024147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38862 (local address 127.2.238.130:42905)
0811 04:39:07.024511 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.024523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.024547 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:39:07.024623 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.024628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.024835 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.025251 (+ 416us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:07.025429 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.026470 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.028002 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.028856 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.029909 (+ 1053us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.030091 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.030143 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.030248 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":136,"threads_started":1}
I20260811 04:39:07.030408 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.024769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32892 (local address 127.2.238.131:33853)
0811 04:39:07.025091 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.025102 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.025124 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:39:07.025205 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.025210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.025389 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.025539 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:07.025762 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.027102 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.028361 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.029115 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.029905 (+ 790us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.030104 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.030161 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.030248 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":109,"threads_started":1}
I20260811 04:39:07.031416 18484 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: No bootstrap required, opened a new log
I20260811 04:39:07.031515 18484 ts_tablet_manager.cc:1397] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260811 04:39:07.031967 18484 raft_consensus.cc:348] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.032092 18484 raft_consensus.cc:374] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:07.032130 18484 raft_consensus.cc:729] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b276ea40644b3f92ee38c84668651b, State: Initialized, Role: FOLLOWER
I20260811 04:39:07.032305 18484 consensus_queue.cc:260] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.032590 18484 ts_tablet_manager.cc:1428] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260811 04:39:07.033190 18820 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 (DEFAULT_TABLE table=test-table [id=f929e7b861fc499491a1e7806fb3c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:07.033172 18679 tablet_service.cc:1467] Processing CreateTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 (DEFAULT_TABLE table=test-table [id=f929e7b861fc499491a1e7806fb3c51c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:07.034266 18820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:07.034448 18679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:07.041163 18904 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Bootstrap starting.
I20260811 04:39:07.041203 18905 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Bootstrap starting.
I20260811 04:39:07.043640 18904 tablet_bootstrap.cc:654] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:07.044027 18905 tablet_bootstrap.cc:654] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:07.044497 18904 log.cc:824] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:07.044852 18905 log.cc:824] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:07.046489 18904 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: No bootstrap required, opened a new log
I20260811 04:39:07.046602 18905 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: No bootstrap required, opened a new log
I20260811 04:39:07.046725 18904 ts_tablet_manager.cc:1397] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:39:07.046839 18905 ts_tablet_manager.cc:1397] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:39:07.049479 18905 raft_consensus.cc:348] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.049479 18904 raft_consensus.cc:348] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.049726 18905 raft_consensus.cc:374] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:07.049734 18904 raft_consensus.cc:374] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:07.049790 18904 raft_consensus.cc:729] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dd85a26d1564aa2816ee7b790cfcebf, State: Initialized, Role: FOLLOWER
I20260811 04:39:07.049844 18905 raft_consensus.cc:729] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c4109df9aa84f8ba59f625dbfe8d686, State: Initialized, Role: FOLLOWER
I20260811 04:39:07.050350 18905 consensus_queue.cc:260] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.050400 18904 consensus_queue.cc:260] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.050989 18891 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
I20260811 04:39:07.050998 18749 heartbeater.cc:503] Master 127.2.238.190:39151 was elected leader, sending a full tablet report...
I20260811 04:39:07.051342 18904 ts_tablet_manager.cc:1428] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260811 04:39:07.051342 18905 ts_tablet_manager.cc:1428] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260811 04:39:07.053436 18750 tablet.cc:2369] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:39:07.091312 18601 tablet.cc:2369] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:39:07.234127 18892 tablet.cc:2369] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:07.263242 18910 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:07.263605 18910 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.264940 18910 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853), d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491)
I20260811 04:39:07.268994 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.265246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:40495 (local address 127.2.238.131:33853)
0811 04:39:07.265392 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.265399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.265417 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:39:07.265481 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.265483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.265522 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.265619 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:07.265984 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.266668 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.267872 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.268457 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.268545 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.268587 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.268836 (+ 249us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.268893 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:39:07.269008 18751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.265148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:33853 (local address 127.2.238.130:40495)
0811 04:39:07.265257 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.265272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:07.265362 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:07.265688 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:07.265691 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:07.265708 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:07.265866 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.265872 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.266780 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.266783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.267741 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.267750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.268572 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.268576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.268706 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.268732 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:39:07.268809 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 04:39:07.268878 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260811 04:39:07.269593 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.265152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34491 (local address 127.2.238.130:37341)
0811 04:39:07.265729 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.265745 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:07.265895 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:07.266224 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:07.266227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:07.266245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:07.266358 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.266363 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.267279 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.267283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.268225 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:07.268236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.269122 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.269128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:07.269261 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.269292 (+ 31us) client_negotiation.cc:769] Sending connection context
0811 04:39:07.269377 (+ 85us) client_negotiation.cc:240] Negotiation successful
0811 04:39:07.269457 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":286,"thread_start_us":67,"threads_started":1}
I20260811 04:39:07.269626 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.265298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:37341 (local address 127.2.238.129:34491)
0811 04:39:07.265584 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.265589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.265861 (+ 272us) server_negotiation.cc:407] Connection header received
0811 04:39:07.266024 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.266032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.266093 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.266188 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:07.266489 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.267163 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.268364 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.268985 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.269106 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.269267 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.269407 (+ 140us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.269490 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":86,"threads_started":1}
I20260811 04:39:07.269804 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2eea7aca474ea2b6f1945cc3ffcc11" candidate_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" is_pre_election: true
I20260811 04:39:07.270201 18841 raft_consensus.cc:2454] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd85a26d1564aa2816ee7b790cfcebf in term 0.
I20260811 04:39:07.270224 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2eea7aca474ea2b6f1945cc3ffcc11" candidate_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b276ea40644b3f92ee38c84668651b" is_pre_election: true
I20260811 04:39:07.270476 18552 raft_consensus.cc:2454] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3dd85a26d1564aa2816ee7b790cfcebf in term 0.
I20260811 04:39:07.270680 18626 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686; no voters:
I20260811 04:39:07.270944 18910 raft_consensus.cc:2781] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:39:07.271024 18910 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:39:07.271054 18910 raft_consensus.cc:3037] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:07.272094 18910 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.272441 18910 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [CANDIDATE]: Term 1 election: Requested vote from peers 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853), d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491)
I20260811 04:39:07.272770 18552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2eea7aca474ea2b6f1945cc3ffcc11" candidate_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b276ea40644b3f92ee38c84668651b"
I20260811 04:39:07.272774 18841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f2eea7aca474ea2b6f1945cc3ffcc11" candidate_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
I20260811 04:39:07.272897 18841 raft_consensus.cc:3037] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:07.272897 18552 raft_consensus.cc:3037] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:07.274194 18552 raft_consensus.cc:2454] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd85a26d1564aa2816ee7b790cfcebf in term 1.
I20260811 04:39:07.274194 18841 raft_consensus.cc:2454] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3dd85a26d1564aa2816ee7b790cfcebf in term 1.
I20260811 04:39:07.274503 18626 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686, d5b276ea40644b3f92ee38c84668651b; no voters:
I20260811 04:39:07.274696 18910 raft_consensus.cc:2781] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:07.274891 18910 raft_consensus.cc:686] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 LEADER]: Becoming Leader. State: Replica: 3dd85a26d1564aa2816ee7b790cfcebf, State: Running, Role: LEADER
I20260811 04:39:07.275199 18910 consensus_queue.cc:237] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.278470 18402 catalog_manager.cc:5495] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130). New cstate: current_term: 1 leader_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } health_report { overall_health: UNKNOWN } } }
I20260811 04:39:07.325138 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.322055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38876 (local address 127.2.238.130:42905)
0811 04:39:07.322196 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:07.322199 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:07.322211 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:39:07.322287 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:07.322289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:07.322346 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:07.322437 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:07.322648 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.323124 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.323830 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:07.324020 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:07.324091 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:07.324377 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:07.324802 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:39:07.324900 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:07.324976 (+ 76us) server_negotiation.cc:299] Negotiation successful
0811 04:39:07.325026 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:39:07.331107 18841 raft_consensus.cc:1261] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Refusing update from remote peer 3dd85a26d1564aa2816ee7b790cfcebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:07.331104 18552 raft_consensus.cc:1261] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Refusing update from remote peer 3dd85a26d1564aa2816ee7b790cfcebf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:07.331802 18914 consensus_queue.cc:1035] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:07.332000 18910 consensus_queue.cc:1035] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:07.342000 18918 mvcc.cc:205] Tried to move back new op lower bound from 7317189211460653056 to 7317189211242078208. Current Snapshot: MvccSnapshot[applied={T|T < 7317189211460653056}]
I20260811 04:39:07.346935 18398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:07.347100 18398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:07.349326 18398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Sending DeleteTablet for 3 replicas of tablet 7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:07.350001 18530 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:07 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:42750
I20260811 04:39:07.350041 18820 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:07 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:32892
I20260811 04:39:07.350394 18927 tablet_replica.cc:330] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: stopping tablet replica
I20260811 04:39:07.350399 18679 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:07 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:38862
I20260811 04:39:07.350519 18927 raft_consensus.cc:2229] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:07.350639 18927 raft_consensus.cc:2258] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:07.350795 18929 tablet_replica.cc:330] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: stopping tablet replica
I20260811 04:39:07.351095 18929 raft_consensus.cc:2229] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:39:07.351429 18928 tablet_replica.cc:330] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: stopping tablet replica
I20260811 04:39:07.351460 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18441
I20260811 04:39:07.352699 18928 raft_consensus.cc:2229] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:07.351507 18929 raft_consensus.cc:2258] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:07.353147 18928 raft_consensus.cc:2258] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:07.354518 18928 ts_tablet_manager.cc:1905] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:07.354518 18929 ts_tablet_manager.cc:1905] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:07.357163 18929 ts_tablet_manager.cc:1918] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:07.357163 18928 ts_tablet_manager.cc:1918] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:07.357245 18929 log.cc:1196] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:07.357254 18928 log.cc:1196] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:07.357558 18928 ts_tablet_manager.cc:1939] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 6c4109df9aa84f8ba59f625dbfe8d686: Deleting consensus metadata
I20260811 04:39:07.357558 18929 ts_tablet_manager.cc:1939] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P 3dd85a26d1564aa2816ee7b790cfcebf: Deleting consensus metadata
I20260811 04:39:07.358654 18385 catalog_manager.cc:4841] TS 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 (table test-table [id=f929e7b861fc499491a1e7806fb3c51c]) successfully deleted
I20260811 04:39:07.362103 18387 catalog_manager.cc:4841] TS 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): tablet 7f2eea7aca474ea2b6f1945cc3ffcc11 (table test-table [id=f929e7b861fc499491a1e7806fb3c51c]) successfully deleted
W20260811 04:39:07.368928 18387 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:34491 (error 108)
I20260811 04:39:07.369282 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18605
I20260811 04:39:07.370136 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.369820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:34491 (local address 127.0.0.1:42762)
0811 04:39:07.369950 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.370027 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 04:39:07.370576 18387 catalog_manager.cc:4584] TS d5b276ea40644b3f92ee38c84668651b (127.2.238.129:34491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f2eea7aca474ea2b6f1945cc3ffcc11: Network error: Client connection negotiation failed: client connection to 127.2.238.129:34491: connect: Connection refused (error 111)
I20260811 04:39:07.376376 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18752
I20260811 04:39:07.383448 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 18367
I20260811 04:39:07.391424 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--webserver_interface=127.2.238.190
--webserver_port=44619
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:07.510560 18931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:07.510876 18931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:07.510954 18931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:07.515098 18931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:07.515187 18931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:07.515228 18931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:07.515290 18931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:07.515334 18931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:07.521447 18931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=44619
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:07.522722 18931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:07.523814 18931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:07.533119 18937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:07.533073 18939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:07.533182 18931 server_base.cc:1034] running on GCE node
W20260811 04:39:07.533069 18936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:07.534003 18931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:07.535101 18931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:07.536321 18931 hybrid_clock.cc:648] HybridClock initialized: now 1786423147536297 us; error 37 us; skew 500 ppm
Aug 11 04:39:07 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:07.539588 18931 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:07.540966 18931 webserver.cc:460] Webserver started at http://127.2.238.190:44619/ using document root <none> and password file <none>
I20260811 04:39:07.541834 18931 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:07.541952 18931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:07.546414 18931 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:07.548769 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:07.550093 18931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:39:07.550223 18931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "769a46e9dcd348a68dacaa9b0edd1546"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "0a51bf114d9d0da6ea9a99b5a4805b34"
server_key_iv: "4df0560976a36ad8249d226cdbcbd1ee"
server_key_version: "encryptionkey@0"
I20260811 04:39:07.550797 18931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:07.574364 18931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:07.577210 18931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:07.577438 18931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:07.584844 18931 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39151
I20260811 04:39:07.585144 19005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39151 every 8 connection(s)
I20260811 04:39:07.585969 18931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:07.588284 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 18931
I20260811 04:39:07.588737 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:34491
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=35575
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:07.590487 19006 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:39:07.592875 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap starting.
I20260811 04:39:07.618307 19006 log.cc:824] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:07.632872 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:07.633284 19006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: Bootstrap complete.
I20260811 04:39:07.639143 19006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:07.639643 19006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Initialized, Role: FOLLOWER
I20260811 04:39:07.640234 19006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:07.640419 19006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:07.640489 19006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:07.640590 19006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 2 FOLLOWER]: Advancing to term 3
I20260811 04:39:07.643074 19006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:07.643425 19006 leader_election.cc:302] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 769a46e9dcd348a68dacaa9b0edd1546; no voters:
I20260811 04:39:07.643790 19006 leader_election.cc:288] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 04:39:07.643973 19010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 04:39:07.644394 19010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [term 3 LEADER]: Becoming Leader. State: Replica: 769a46e9dcd348a68dacaa9b0edd1546, State: Running, Role: LEADER
I20260811 04:39:07.644765 19010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } }
I20260811 04:39:07.645049 19006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:07.646194 19012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 769a46e9dcd348a68dacaa9b0edd1546. Latest consensus state: current_term: 3 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:07.646324 19012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:07.646981 19011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "769a46e9dcd348a68dacaa9b0edd1546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "769a46e9dcd348a68dacaa9b0edd1546" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39151 } } }
I20260811 04:39:07.647078 19011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:07.650300 19029 catalog_manager.cc:1259] Loaded cluster ID: 8f2ef170694c4462b0526db8c0ee86dd
I20260811 04:39:07.650350 19029 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: loading cluster ID for follower catalog manager: success
W20260811 04:39:07.651386 19029 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 769a46e9dcd348a68dacaa9b0edd1546: acquiring CA information for follower catalog manager: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
W20260811 04:39:07.651463 19029 catalog_manager.cc:873] Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO: failed to prepare follower catalog manager, will retry
I20260811 04:39:07.651621 19020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:07.655408 19020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:07.655911 19020 catalog_manager.cc:1259] Loaded cluster ID: 8f2ef170694c4462b0526db8c0ee86dd
I20260811 04:39:07.655968 19020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260811 04:39:07.656554 19020 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f48744f99ad google::LogMessage::Fail() at ??:0
@ 0x7f48744fc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f48744f94de google::LogMessage::Flush() at ??:0
@ 0x7f48744faa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f487ee38fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f487ee39d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f487ee36579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f487ee7a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f487674b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f487674bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f487674d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f487673c512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4877d806db start_thread at ??:0
@ 0x7f4872ff761f clone at ??:0
W20260811 04:39:07.740260 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:07.740545 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:07.740633 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:07.744668 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:07.744853 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:07.744971 19008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:39:07.750371 19008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:34491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=35575
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:07.751390 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:07.752460 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:07.761914 19050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:07.761926 19051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:07.762382 19053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:07.762754 19008 server_base.cc:1034] running on GCE node
I20260811 04:39:07.763259 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:07.763922 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:07.765138 19008 hybrid_clock.cc:648] HybridClock initialized: now 1786423147765115 us; error 42 us; skew 500 ppm
Aug 11 04:39:07 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:07.768258 19008 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:39:07.769224 19008 webserver.cc:460] Webserver started at http://127.2.238.129:35575/ using document root <none> and password file <none>
I20260811 04:39:07.769869 19008 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:07.769956 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:07.774087 19008 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260811 04:39:07.776491 19062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:07.777940 19008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:39:07.778100 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "d5b276ea40644b3f92ee38c84668651b"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "1c03d4244447ae2750281b0acda50313"
server_key_iv: "dafb72446e8b6e6e7a498e95accd6f70"
server_key_version: "encryptionkey@0"
I20260811 04:39:07.778672 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:07.809126 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:07.812728 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:07.812948 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:07.813652 19008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:07.814901 19069 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 04:39:07.819149 19008 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 04:39:07.819213 19008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260811 04:39:07.819308 19008 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 04:39:07.821151 19008 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 04:39:07.821210 19008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 04:39:07.821480 19069 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Bootstrap starting.
I20260811 04:39:07.831956 19008 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:34491
I20260811 04:39:07.833189 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:34491 every 8 connection(s)
I20260811 04:39:07.833303 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:39:07.837733 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19008
I20260811 04:39:07.838232 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:42905
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=43051
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:07.843115 19069 log.cc:824] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:07.850351 19069 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:07.850734 19069 tablet_bootstrap.cc:492] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Bootstrap complete.
I20260811 04:39:07.851868 19069 ts_tablet_manager.cc:1397] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.015s
I20260811 04:39:07.854211 19069 raft_consensus.cc:348] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.854637 19069 raft_consensus.cc:729] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b276ea40644b3f92ee38c84668651b, State: Initialized, Role: FOLLOWER
I20260811 04:39:07.855167 19069 consensus_queue.cc:260] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:07.856127 19069 ts_tablet_manager.cc:1428] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
W20260811 04:39:07.867213 19187 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:39:07.835907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:54273)
0811 04:39:07.842168 (+ 6261us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.842237 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:07.843328 (+ 1091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:07.866380 (+ 23052us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5975,"thread_start_us":151,"threads_started":1}
W20260811 04:39:07.867893 19185 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39151 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39151: Client connection negotiation failed: client connection to 127.2.238.190:39151: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 04:39:07.868763 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.868490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:40517)
0811 04:39:07.868639 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.868708 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260811 04:39:07.869414 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:07.869205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:49445)
0811 04:39:07.869320 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:07.869368 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260811 04:39:07.869585 19185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:39:07.975337 19188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:07.975625 19188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:07.975709 19188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:07.979859 19188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:07.979949 19188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:07.980036 19188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:39:07.985507 19188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:42905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=43051
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:07.986676 19188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:07.987756 19188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:07.997572 19195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:07.997594 19198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:07.997619 19196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:07.997983 19188 server_base.cc:1034] running on GCE node
I20260811 04:39:07.998377 19188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:07.998986 19188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:08.000275 19188 hybrid_clock.cc:648] HybridClock initialized: now 1786423148000242 us; error 47 us; skew 500 ppm
Aug 11 04:39:08 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:08.003961 19188 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:39:08.005214 19188 webserver.cc:460] Webserver started at http://127.2.238.130:43051/ using document root <none> and password file <none>
I20260811 04:39:08.006237 19188 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:08.006493 19188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:08.011977 19188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 04:39:08.014148 19205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.015447 19188 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:39:08.015587 19188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "3dd85a26d1564aa2816ee7b790cfcebf"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "23b190482d7917d7762fcb8c7b6a34e5"
server_key_iv: "4568debec1c2548aa68d77a20a092ab1"
server_key_version: "encryptionkey@0"
I20260811 04:39:08.016144 19188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:08.030130 19188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:08.032789 19188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:08.032989 19188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:08.033567 19188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:08.034422 19188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:08.034466 19188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.034530 19188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:08.034561 19188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.043529 19188 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:42905
I20260811 04:39:08.043571 19324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:42905 every 8 connection(s)
I20260811 04:39:08.044587 19188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:39:08.048485 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.046783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:45431)
0811 04:39:08.047209 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.047309 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":113,"threads_started":1}
W20260811 04:39:08.048981 19325 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39151 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39151: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
I20260811 04:39:08.049124 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19188
I20260811 04:39:08.049651 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.049395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:51599)
0811 04:39:08.049522 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.049583 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260811 04:39:08.049633 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:33853
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:39151
--webserver_port=45665
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:08.050418 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.050152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:40339)
0811 04:39:08.050282 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.050343 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 04:39:08.050604 19325 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:39:08.161113 19328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:08.161372 19328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:08.161463 19328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:08.165158 19328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:08.165248 19328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:08.165337 19328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:39:08.170324 19328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:33853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=45665
--tserver_master_addrs=127.2.238.190:39151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:08.171316 19328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:08.172267 19328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:08.181533 19334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:08.181641 19328 server_base.cc:1034] running on GCE node
W20260811 04:39:08.181587 19333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:08.181565 19336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:08.182279 19328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:08.182924 19328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:08.184104 19328 hybrid_clock.cc:648] HybridClock initialized: now 1786423148184073 us; error 42 us; skew 500 ppm
Aug 11 04:39:08 dist-test-slave-0kls krb5kdc[17792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423148, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:08.186614 19328 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:39:08.187429 19328 webserver.cc:460] Webserver started at http://127.2.238.131:45665/ using document root <none> and password file <none>
I20260811 04:39:08.187984 19328 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:08.188076 19328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:08.191985 19328 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:39:08.194205 19344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.195402 19328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:08.195542 19328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "6c4109df9aa84f8ba59f625dbfe8d686"
format_stamp: "Formatted at 2026-08-11 04:39:05 on dist-test-slave-0kls"
server_key: "6ee2632780e0eec287d9a2185e88ab2a"
server_key_iv: "4c2f65c031784cc4248e2d42aa562cb7"
server_key_version: "encryptionkey@0"
I20260811 04:39:08.196101 19328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:08.206634 19328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:08.209312 19328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:08.209509 19328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:08.210199 19328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:08.211037 19328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:08.211110 19328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.211175 19328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:08.211223 19328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:08.220969 19328 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:33853
I20260811 04:39:08.221032 19466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:33853 every 8 connection(s)
I20260811 04:39:08.222028 19328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:39:08.225771 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.224123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:36013)
0811 04:39:08.224569 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.224671 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":127,"threads_started":1}
W20260811 04:39:08.226397 19467 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39151 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39151: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
I20260811 04:39:08.227001 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19328
I20260811 04:39:08.227043 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.226823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:43633)
0811 04:39:08.226940 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.226988 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 04:39:08.227761 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.227531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:53917)
0811 04:39:08.227641 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.227694 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260811 04:39:08.228103 19467 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 04:39:08.870661 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:08.870207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:54181)
0811 04:39:08.870474 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:08.870537 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":71,"threads_started":1}
I20260811 04:39:09.051537 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.051143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:53105)
0811 04:39:09.051379 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:09.051431 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260811 04:39:09.229069 19474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.228683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:56215)
0811 04:39:09.228924 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:09.228977 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260811 04:39:09.245379 19475 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:09.245556 19475 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:09.246645 19475 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
Aug 11 04:39:09 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.131@KRBTEST.COM
Aug 11 04:39:09 dist-test-slave-0kls krb5kdc[17792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423147, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.130@KRBTEST.COM
I20260811 04:39:09.260754 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.246922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:42905 (local address 127.2.238.129:51907)
0811 04:39:09.247114 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:09.247130 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:09.247238 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:09.248970 (+ 1732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:09.248976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:09.249555 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:09.250112 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:09.250127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.251619 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.251625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:09.252500 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:09.252510 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.253230 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.253235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:09.253450 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:09.254290 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:09.254305 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:09.256227 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:09.258618 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:09.258621 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:09.258623 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:09.258884 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:09.259225 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:09.259228 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:09.259230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:09.259351 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:09.259956 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:09.259959 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:09.260068 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 04:39:09.260248 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 04:39:09.260556 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":141,"spinlock_wait_cycles":22528}
I20260811 04:39:09.260771 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.247051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:49695 (local address 127.2.238.131:33853)
0811 04:39:09.247535 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:09.247558 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:09.247602 (+ 44us) server_negotiation.cc:407] Connection header received
0811 04:39:09.248284 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:09.248290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:09.248606 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:09.248920 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:09.250326 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.251470 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.252620 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.253064 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.253252 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:09.256104 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:09.256111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:09.256128 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:09.256160 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:09.258015 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:09.258626 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:09.258633 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:09.258638 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:09.258793 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:09.259225 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:09.259228 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:09.259230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:09.259688 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:09.259812 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:09.260364 (+ 552us) server_negotiation.cc:299] Negotiation successful
0811 04:39:09.260610 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":128,"threads_started":1}
I20260811 04:39:09.260838 19478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.247186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:51907 (local address 127.2.238.130:42905)
0811 04:39:09.247520 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:09.247550 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:09.247638 (+ 88us) server_negotiation.cc:407] Connection header received
0811 04:39:09.248338 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:09.248344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:09.248657 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:09.248970 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:09.250286 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.251487 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.252626 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.253074 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.253282 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:09.256443 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:09.256447 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:09.256457 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:09.256482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:09.258514 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:09.258989 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:09.258995 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:09.259000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:09.259120 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:09.259444 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:09.259447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:09.259448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:09.259847 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:09.259969 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:09.260357 (+ 388us) server_negotiation.cc:299] Negotiation successful
0811 04:39:09.260645 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":62,"threads_started":1}
I20260811 04:39:09.260885 19477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.246922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:33853 (local address 127.2.238.129:49695)
0811 04:39:09.247464 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:09.247477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:09.247566 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:09.248968 (+ 1402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:09.248974 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:09.249555 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:09.250111 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:09.250127 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.251609 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.251612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:09.252498 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:09.252507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:09.253205 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:09.253208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:09.253449 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:09.254217 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:09.254233 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:09.255877 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:09.258154 (+ 2277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:09.258159 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:09.258172 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:09.258493 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:09.258913 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:09.258915 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:09.258916 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:09.259116 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:09.259824 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:09.259830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:09.260086 (+ 256us) client_negotiation.cc:769] Sending connection context
0811 04:39:09.260248 (+ 162us) client_negotiation.cc:240] Negotiation successful
0811 04:39:09.260645 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":473,"spinlock_wait_cycles":10368,"thread_start_us":78,"threads_started":1}
W20260811 04:39:09.263528 19063 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
W20260811 04:39:09.263706 19065 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:09.263813 19065 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b276ea40644b3f92ee38c84668651b; no voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686
I20260811 04:39:09.264122 19475 raft_consensus.cc:2726] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:39:09.871896 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:09.871489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:60967)
0811 04:39:09.871738 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:09.871797 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":78,"threads_started":1}
I20260811 04:39:10.052680 19480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:10.052260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:47405)
0811 04:39:10.052535 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:10.052595 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260811 04:39:10.230217 19481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:10.229741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:39427)
0811 04:39:10.230074 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:10.230131 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":151,"threads_started":1}
I20260811 04:39:10.873193 19482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:10.872668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:55861)
0811 04:39:10.873038 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:10.873095 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":74,"threads_started":1}
I20260811 04:39:11.053915 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:11.053486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:45391)
0811 04:39:11.053716 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:11.053765 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
I20260811 04:39:11.231282 19485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:11.230891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:50145)
0811 04:39:11.231135 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:11.231184 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":69,"threads_started":1}
I20260811 04:39:11.317261 19486 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:11.317390 19486 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:11.317876 19486 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
W20260811 04:39:11.318725 19063 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
W20260811 04:39:11.318809 19065 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:11.318849 19063 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b276ea40644b3f92ee38c84668651b; no voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686
I20260811 04:39:11.319016 19486 raft_consensus.cc:2726] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:39:11.874512 19487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:11.874080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:54499)
0811 04:39:11.874354 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:11.874413 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":88,"threads_started":1}
I20260811 04:39:12.055081 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:12.054659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:36169)
0811 04:39:12.054916 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:12.054984 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":92,"threads_started":1}
I20260811 04:39:12.232507 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:12.232121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:42477)
0811 04:39:12.232366 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:12.232418 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260811 04:39:12.876235 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:12.875667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:50681)
0811 04:39:12.876070 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:12.876134 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":131,"threads_started":1}
I20260811 04:39:13.056370 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:13.055841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:57745)
0811 04:39:13.056156 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:13.056249 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":72,"threads_started":1}
I20260811 04:39:13.233803 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:13.233320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:39987)
0811 04:39:13.233618 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:13.233683 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
I20260811 04:39:13.877441 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:13.877037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:52913)
0811 04:39:13.877289 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:13.877344 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":74,"threads_started":1}
I20260811 04:39:14.057603 19494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:14.057186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:47463)
0811 04:39:14.057461 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:14.057516 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
I20260811 04:39:14.234908 19495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:14.234511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:32939)
0811 04:39:14.234760 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:14.234821 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260811 04:39:14.615522 19496 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:14.615653 19496 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:14.616161 19496 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
W20260811 04:39:14.616962 19065 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
W20260811 04:39:14.617065 19063 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:14.617137 19063 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b276ea40644b3f92ee38c84668651b; no voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686
I20260811 04:39:14.617269 19496 raft_consensus.cc:2726] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:39:14.878780 19497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:14.878292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:49727)
0811 04:39:14.878615 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:14.878681 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":97,"threads_started":1}
I20260811 04:39:15.058785 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:15.058341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:37171)
0811 04:39:15.058638 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:15.058693 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":73,"threads_started":1}
I20260811 04:39:15.236305 19501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:15.235730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:38345)
0811 04:39:15.236128 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:15.236193 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":120,"threads_started":1}
I20260811 04:39:15.879987 19502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:15.879533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:41823)
0811 04:39:15.879831 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:15.879888 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":95,"threads_started":1}
I20260811 04:39:16.060063 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:16.059617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:35801)
0811 04:39:16.059912 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:16.059968 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":76,"threads_started":1}
I20260811 04:39:16.237596 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:16.237139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:41543)
0811 04:39:16.237407 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:16.237484 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":85,"threads_started":1}
I20260811 04:39:16.881258 19505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:16.880832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:51863)
0811 04:39:16.881092 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:16.881155 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260811 04:39:17.061391 19507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:17.060956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:50729)
0811 04:39:17.061248 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:17.061301 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260811 04:39:17.238878 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:17.238436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:38885)
0811 04:39:17.238727 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:17.238781 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":110,"threads_started":1}
I20260811 04:39:17.882767 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:17.882293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:44639)
0811 04:39:17.882555 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:17.882613 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260811 04:39:18.062714 19510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:18.062247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:37675)
0811 04:39:18.062549 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:18.062615 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":78,"threads_started":1}
I20260811 04:39:18.240156 19511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:18.239702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:34999)
0811 04:39:18.239999 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:18.240058 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":124,"threads_started":1}
I20260811 04:39:18.683669 19512 raft_consensus.cc:482] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:18.683825 19512 raft_consensus.cc:504] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3dd85a26d1564aa2816ee7b790cfcebf" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42905 } } peers { permanent_uuid: "6c4109df9aa84f8ba59f625dbfe8d686" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 33853 } } peers { permanent_uuid: "d5b276ea40644b3f92ee38c84668651b" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 34491 } }
I20260811 04:39:18.684281 19512 leader_election.cc:288] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905), 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853)
W20260811 04:39:18.685017 19065 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3dd85a26d1564aa2816ee7b790cfcebf (127.2.238.130:42905): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
W20260811 04:39:18.685148 19063 leader_election.cc:341] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 6c4109df9aa84f8ba59f625dbfe8d686 (127.2.238.131:33853): Not found: Tablet not found: 7f2eea7aca474ea2b6f1945cc3ffcc11
I20260811 04:39:18.685289 19063 leader_election.cc:302] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5b276ea40644b3f92ee38c84668651b; no voters: 3dd85a26d1564aa2816ee7b790cfcebf, 6c4109df9aa84f8ba59f625dbfe8d686
I20260811 04:39:18.685519 19512 raft_consensus.cc:2726] T 7f2eea7aca474ea2b6f1945cc3ffcc11 P d5b276ea40644b3f92ee38c84668651b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:39:18.883976 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:18.883490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:33141)
0811 04:39:18.883820 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:18.883879 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":99,"threads_started":1}
I20260811 04:39:19.063911 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:19.063490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:33397)
0811 04:39:19.063761 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:19.063816 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260811 04:39:19.241403 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:19.240947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:32975)
0811 04:39:19.241249 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:19.241308 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":90,"threads_started":1}
I20260811 04:39:19.885159 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:19.884702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:40519)
0811 04:39:19.884975 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:19.885032 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260811 04:39:20.065272 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:20.064843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:39951)
0811 04:39:20.065116 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:20.065175 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":73,"threads_started":1}
I20260811 04:39:20.242699 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:20.242208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:38635)
0811 04:39:20.242521 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:20.242598 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260811 04:39:20.886488 19519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:20.886039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:36307)
0811 04:39:20.886321 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:20.886385 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":86,"threads_started":1}
I20260811 04:39:21.066509 19521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:21.066039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:43857)
0811 04:39:21.066350 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:21.066410 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":101,"threads_started":1}
I20260811 04:39:21.243920 19522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:21.243488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:50245)
0811 04:39:21.243755 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:21.243823 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260811 04:39:21.887897 19523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:21.887329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:58137)
0811 04:39:21.887706 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:21.887793 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":77,"threads_started":1}
I20260811 04:39:22.067971 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:22.067433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:45619)
0811 04:39:22.067751 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:22.067842 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":94,"threads_started":1}
I20260811 04:39:22.245230 19525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:22.244808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:58439)
0811 04:39:22.245077 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:22.245130 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":98,"threads_started":1}
I20260811 04:39:22.889580 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:22.889021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.129:39787)
0811 04:39:22.889395 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:22.889465 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":120,"threads_started":1}
I20260811 04:39:23.069456 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:23.068971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.130:39873)
0811 04:39:23.069297 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:23.069359 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":102,"threads_started":1}
I20260811 04:39:23.228881 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19008
I20260811 04:39:23.239961 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19188
I20260811 04:39:23.246835 19529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:23.246393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39151 (local address 127.2.238.131:36231)
0811 04:39:23.246675 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:23.246735 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
I20260811 04:39:23.249536 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19328
2026-08-11T04:39:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19473 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:39:23 dist-test-slave-0kls krb5kdc[19532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:39:23 dist-test-slave-0kls krb5kdc[19532](info): set up 2 sockets
Aug 11 04:39:23 dist-test-slave-0kls krb5kdc[19532](info): commencing operation
krb5kdc: starting...
W20260811 04:39:24.306205 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:39:24 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:39:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:39:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:39:24.432909 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:24.543344 19547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:24.543594 19547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:24.543632 19547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:24.547386 19547 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 04:39:24.547431 19547 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 04:39:24.547446 19547 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 04:39:24.547457 19547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:24.547473 19547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:24.547542 19547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:24.547569 19547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:24.547598 19547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:24.552567 19547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:24.553596 19547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:24.554620 19547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:24.563241 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:24.563308 19552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:24.563241 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:24.563316 19547 server_base.cc:1034] running on GCE node
I20260811 04:39:24.563972 19547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:24.564998 19547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:24.566156 19547 hybrid_clock.cc:648] HybridClock initialized: now 1786423164566145 us; error 24 us; skew 500 ppm
Aug 11 04:39:24 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:24.568796 19547 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:24.569619 19547 webserver.cc:460] Webserver started at http://127.2.238.190:43257/ using document root <none> and password file <none>
I20260811 04:39:24.570245 19547 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:24.570307 19547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:24.570537 19547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:24.572492 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "3c3857a3548647dd97d1f896caa8cc19"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "f18b593221a15ddc5f201b5d33dc95d8"
server_key_iv: "20be476ffb77776c11251eba2d5a3316"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.573051 19547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "3c3857a3548647dd97d1f896caa8cc19"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "f18b593221a15ddc5f201b5d33dc95d8"
server_key_iv: "20be476ffb77776c11251eba2d5a3316"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.576592 19547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:39:24.578876 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:24.580093 19547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:39:24.580227 19547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "3c3857a3548647dd97d1f896caa8cc19"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "f18b593221a15ddc5f201b5d33dc95d8"
server_key_iv: "20be476ffb77776c11251eba2d5a3316"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.580327 19547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:24.598228 19547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:24.602280 19547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:24.602489 19547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:24.609082 19547 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39749
I20260811 04:39:24.609365 19621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39749 every 8 connection(s)
I20260811 04:39:24.610069 19547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:24.612744 19622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:24.617887 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap starting.
I20260811 04:39:24.618955 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19547
I20260811 04:39:24.619107 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:39:24.619375 3002 external_mini_cluster.cc:1428] Setting key dba173180b8b77f6750a317719f6bff2
I20260811 04:39:24.620505 19622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:24.621292 19622 log.cc:824] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:24.623521 19622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: No bootstrap required, opened a new log
I20260811 04:39:24.628147 19622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:24.628389 19622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:24.628456 19622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Initialized, Role: FOLLOWER
Aug 11 04:39:24 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:24.628918 19622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:24.629081 19622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:24.629148 19622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:24.629245 19622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:24.630486 19622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:24.630865 19622 leader_election.cc:302] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c3857a3548647dd97d1f896caa8cc19; no voters:
I20260811 04:39:24.631250 19622 leader_election.cc:288] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:39:24.631390 19627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:24.631676 19627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 LEADER]: Becoming Leader. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Running, Role: LEADER
I20260811 04:39:24.632078 19627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:24.632551 19622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:24.633826 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:39:24.633955 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:24.634214 19628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c3857a3548647dd97d1f896caa8cc19. Latest consensus state: current_term: 1 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:39:24.634290 19628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:24.634965 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:24.620484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.2.238.190:39749)
0811 04:39:24.620927 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:24.620943 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:24.620977 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:39:24.621683 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:24.621707 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:24.622152 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:24.622419 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:24.623296 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.624272 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.625463 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.626231 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.626362 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:24.629104 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:24.629111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:24.629127 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:24.629159 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:24.631200 (+ 2041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:24.631786 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:24.631793 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:24.631801 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:24.631949 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:24.632397 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:24.632401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:24.632402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:24.632910 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:24.633116 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:24.633528 (+ 412us) server_negotiation.cc:299] Negotiation successful
0811 04:39:24.633910 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":156,"threads_started":1}
W20260811 04:39:24.637749 19642 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:39:24.637892 19642 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:39:24.638041 19636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:24.638944 19636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:24.644285 19636 catalog_manager.cc:1347] Generated new cluster ID: 2adcdef53c674128bc224b509c12594a
I20260811 04:39:24.644347 19636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:24.652318 19636 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:39:24.653195 19636 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:24.658771 19636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Generated new TSK 0
I20260811 04:39:24.659353 19636 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:39:24.693255 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:24.806285 19648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:24.806541 19648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:24.806582 19648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:24.810443 19648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:24.810495 19648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:24.810585 19648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:39:24.815580 19648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:24.816633 19648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:24.817636 19648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:24.827191 19655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:24.827215 19657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:24.827215 19654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:24.827529 19648 server_base.cc:1034] running on GCE node
I20260811 04:39:24.827898 19648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:24.828470 19648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:24.829627 19648 hybrid_clock.cc:648] HybridClock initialized: now 1786423164829610 us; error 29 us; skew 500 ppm
Aug 11 04:39:24 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:24.832443 19648 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:39:24.833377 19648 webserver.cc:460] Webserver started at http://127.2.238.129:44815/ using document root <none> and password file <none>
I20260811 04:39:24.834029 19648 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:24.834100 19648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:24.834343 19648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:24.836516 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "c6464be8500143359c73bbb30fb7b704"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "a87ade3e84145568387e9fc4930ef383"
server_key_iv: "2e672580f58633b0a65d5f3ecf4e4d20"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.837083 19648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "c6464be8500143359c73bbb30fb7b704"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "a87ade3e84145568387e9fc4930ef383"
server_key_iv: "2e672580f58633b0a65d5f3ecf4e4d20"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.840793 19648 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:39:24.843201 19665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:24.844396 19648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:39:24.844535 19648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "c6464be8500143359c73bbb30fb7b704"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "a87ade3e84145568387e9fc4930ef383"
server_key_iv: "2e672580f58633b0a65d5f3ecf4e4d20"
server_key_version: "encryptionkey@0"
I20260811 04:39:24.844636 19648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:24.859494 19648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:24.862741 19648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:24.862939 19648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:24.863528 19648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:24.864331 19648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:24.864382 19648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:24.864465 19648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:24.864496 19648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:24.873409 19648 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44585
I20260811 04:39:24.873435 19785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44585 every 8 connection(s)
I20260811 04:39:24.874423 19648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:39:24.880378 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19648
I20260811 04:39:24.880517 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:39:24.880821 3002 external_mini_cluster.cc:1428] Setting key 8250f414ae3e7f421254b5eeb924d9a9
Aug 11 04:39:24 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423164, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:24.888206 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:24.876236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56275 (local address 127.2.238.190:39749)
0811 04:39:24.876378 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:24.876382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:24.877334 (+ 952us) server_negotiation.cc:407] Connection header received
0811 04:39:24.878169 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:24.878172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:24.878219 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:24.878289 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:24.879491 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.880026 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.881335 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.881656 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.881761 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:24.884726 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:24.884730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:24.884733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:24.884766 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:24.886174 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:24.886690 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:24.886694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:24.886695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:24.886770 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:24.887105 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:24.887108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:24.887110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:24.887295 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:24.887400 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:24.887924 (+ 524us) server_negotiation.cc:299] Negotiation successful
0811 04:39:24.888044 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:39:24.889077 19788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:24.876541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:56275)
0811 04:39:24.877197 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:24.877237 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:24.877993 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:24.878460 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:24.878468 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:24.878852 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:24.879355 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:24.879370 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.880134 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.880136 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:24.881200 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:24.881212 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:24.881859 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:24.881864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:24.882038 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:24.882725 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:24.882742 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:24.884506 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:24.886283 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:24.886289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:24.886300 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:24.886574 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:24.886868 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:24.886873 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:24.886874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:24.887008 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:24.887400 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:24.887406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:24.887702 (+ 296us) client_negotiation.cc:769] Sending connection context
0811 04:39:24.887882 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 04:39:24.888184 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":114,"threads_started":1}
I20260811 04:39:24.890390 19786 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:24.890652 19786 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:24.891284 19786 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:39:24.892921 19576 ts_manager.cc:194] Registered new tserver with Master: c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585)
I20260811 04:39:24.894152 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:56275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:39:24.910612 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:25.017683 19791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:25.017973 19791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:25.018042 19791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:25.021773 19791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:25.021881 19791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:25.021971 19791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:39:25.026979 19791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:25.028061 19791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:25.029018 19791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:25.038561 19797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:25.038709 19791 server_base.cc:1034] running on GCE node
W20260811 04:39:25.038566 19796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:25.038867 19800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:25.039315 19791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:25.039963 19791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:25.041152 19791 hybrid_clock.cc:648] HybridClock initialized: now 1786423165041120 us; error 42 us; skew 500 ppm
Aug 11 04:39:25 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:25.043991 19791 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:39:25.044811 19791 webserver.cc:460] Webserver started at http://127.2.238.130:35295/ using document root <none> and password file <none>
I20260811 04:39:25.045346 19791 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:25.045405 19791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:25.045569 19791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:25.047797 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "21b8db948e9045aaabeb9467f7031d67"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "456a95b8fd6c8804306b937e479c0790"
server_key_iv: "be558f930a15788c4b9509cf02c607fe"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.048380 19791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "21b8db948e9045aaabeb9467f7031d67"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "456a95b8fd6c8804306b937e479c0790"
server_key_iv: "be558f930a15788c4b9509cf02c607fe"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.051986 19791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260811 04:39:25.054225 19808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.055358 19791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:25.055471 19791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "21b8db948e9045aaabeb9467f7031d67"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "456a95b8fd6c8804306b937e479c0790"
server_key_iv: "be558f930a15788c4b9509cf02c607fe"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.055600 19791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:25.064752 19791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:25.067946 19791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:25.068143 19791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:25.068739 19791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:25.069517 19791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:25.069563 19791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.069628 19791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:25.069659 19791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.078286 19930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:41465 every 8 connection(s)
I20260811 04:39:25.078264 19791 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:41465
I20260811 04:39:25.079221 19791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:39:25.087988 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19791
I20260811 04:39:25.088138 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:39:25.088454 3002 external_mini_cluster.cc:1428] Setting key 6f40bf92d746a22e1a41b9546db62dba
Aug 11 04:39:25 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:25.093214 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.081134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:37557 (local address 127.2.238.190:39749)
0811 04:39:25.081284 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.081288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.082360 (+ 1072us) server_negotiation.cc:407] Connection header received
0811 04:39:25.083226 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.083229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.083304 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.083407 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:25.084635 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.085213 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.086531 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.086737 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.086849 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.089378 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:25.089381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:25.089383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:25.089407 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:25.091208 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.091673 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.091679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.091684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.091775 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.092082 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.092086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.092097 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.092329 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:25.092407 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.092985 (+ 578us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.093083 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:39:25.094357 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.081507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:37557)
0811 04:39:25.082183 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.082242 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.083001 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.083583 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.083590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.083964 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:25.084499 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.084512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.085357 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.085360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.086401 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.086426 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.086892 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.086895 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.087047 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.087646 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:25.087661 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:25.089203 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:25.091324 (+ 2121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:25.091329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:25.091339 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:25.091576 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:25.091864 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:25.091866 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:25.091867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:25.091986 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:25.092436 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:25.092444 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:25.092738 (+ 294us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.092953 (+ 215us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.093293 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":110,"threads_started":1}
I20260811 04:39:25.095470 19931 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:25.095711 19931 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:25.096166 19931 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
I20260811 04:39:25.097244 19576 ts_manager.cc:194] Registered new tserver with Master: 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465)
I20260811 04:39:25.097856 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:37557
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:39:25.117290 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:25.229234 19938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:25.229473 19938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:25.229512 19938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:25.233511 19938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:25.233595 19938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:25.233695 19938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:39:25.238626 19938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:25.239616 19938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:25.240653 19938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:25.250109 19944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:25.250146 19947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:25.250173 19943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:25.250311 19938 server_base.cc:1034] running on GCE node
I20260811 04:39:25.250890 19938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:25.251543 19938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:25.252704 19938 hybrid_clock.cc:648] HybridClock initialized: now 1786423165252684 us; error 28 us; skew 500 ppm
Aug 11 04:39:25 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:25.255395 19938 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:39:25.256213 19938 webserver.cc:460] Webserver started at http://127.2.238.131:43523/ using document root <none> and password file <none>
I20260811 04:39:25.256907 19938 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:25.257004 19938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:25.257244 19938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:39:25.259480 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "ac3bc6f24a0746dcabe1a67c6f22f408"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "c7336f2ed7a43a4cdb69f194ffac234a"
server_key_iv: "1638d3150fb55ed434badc8e83a57137"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.260120 19938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "ac3bc6f24a0746dcabe1a67c6f22f408"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "c7336f2ed7a43a4cdb69f194ffac234a"
server_key_iv: "1638d3150fb55ed434badc8e83a57137"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.264014 19938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260811 04:39:25.266562 19955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.267740 19938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:39:25.267911 19938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "ac3bc6f24a0746dcabe1a67c6f22f408"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "c7336f2ed7a43a4cdb69f194ffac234a"
server_key_iv: "1638d3150fb55ed434badc8e83a57137"
server_key_version: "encryptionkey@0"
I20260811 04:39:25.268026 19938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:25.284086 19938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:25.287087 19938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:25.287292 19938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:25.287894 19938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:25.288725 19938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:25.288815 19938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.288882 19938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:25.288914 19938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:25.297550 19938 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:37741
I20260811 04:39:25.297574 20076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:37741 every 8 connection(s)
I20260811 04:39:25.298372 19938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:39:25.304225 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 19938
I20260811 04:39:25.304369 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:39:25.304638 3002 external_mini_cluster.cc:1428] Setting key ed194504fd8e1066f143dbbed5860960
Aug 11 04:39:25 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423165, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:25.311861 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.300115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:48813 (local address 127.2.238.190:39749)
0811 04:39:25.300267 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.300271 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.301250 (+ 979us) server_negotiation.cc:407] Connection header received
0811 04:39:25.302125 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.302128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.302176 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.302248 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:25.303296 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.303820 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.305086 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.305347 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.305458 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.308162 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:25.308166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:25.308168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:25.308225 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:25.309740 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.310336 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.310340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.310342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.310429 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.310778 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.310782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.310784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.310975 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:25.311094 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.311576 (+ 482us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.311702 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 04:39:25.312693 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.300425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:48813)
0811 04:39:25.301084 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.301141 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.301943 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.302372 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.302379 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.302713 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:25.303171 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.303184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.303930 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.303933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.304938 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.304948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.305532 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.305537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.305742 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.306398 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:25.306427 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:25.307987 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:25.309934 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:25.309940 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:25.309955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:25.310225 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:25.310527 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:25.310530 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:25.310532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:25.310676 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:25.311128 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:25.311134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:25.311357 (+ 223us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.311537 (+ 180us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.311839 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":112,"threads_started":1}
I20260811 04:39:25.313779 20077 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:25.314059 20077 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:25.314568 20077 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
I20260811 04:39:25.315557 19576 ts_manager.cc:194] Registered new tserver with Master: ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
I20260811 04:39:25.316111 19576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:48813
I20260811 04:39:25.318638 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:39:25.327353 19625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.320623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.2.238.190:39749)
0811 04:39:25.320740 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.320744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.320920 (+ 176us) server_negotiation.cc:407] Connection header received
0811 04:39:25.321016 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.321018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.321066 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.321146 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:25.321778 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.322314 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.323082 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.323294 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.323374 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.324609 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:25.324612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:25.324614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:25.324634 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:25.325761 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.326281 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.326284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.326286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.326377 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:25.326622 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:25.326624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:25.326625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:25.326833 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:25.326938 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.327128 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.327239 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:39:25.330259 19576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:39:25.333247 19576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:39:25.347872 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.341645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.0.0.1:38496)
0811 04:39:25.342156 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.342201 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.342332 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.343592 (+ 1260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.343595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.343612 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:25.343717 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.343726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.345149 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.345152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.346267 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.346274 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.347191 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.347194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.347348 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347477 (+ 129us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.347580 (+ 103us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.347726 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":186,"thread_start_us":97,"threads_started":1}
I20260811 04:39:25.347894 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.342536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38496 (local address 127.2.238.129:44585)
0811 04:39:25.343167 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.343182 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.343204 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:39:25.343280 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.343285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.343461 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.343613 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:25.343870 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.345006 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.346374 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.347081 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.347190 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347328 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.347631 (+ 303us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.347734 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":144,"threads_started":1}
I20260811 04:39:25.348039 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.341984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.2.238.130:41465)
0811 04:39:25.342432 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.342446 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.342898 (+ 452us) server_negotiation.cc:407] Connection header received
0811 04:39:25.342992 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.342998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.343209 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.343357 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:25.343849 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.344604 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.346208 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.346853 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.346973 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347079 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.347748 (+ 669us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.347844 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20260811 04:39:25.348359 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.342373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:37741 (local address 127.0.0.1:54388)
0811 04:39:25.343106 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.343118 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.343212 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.343844 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.343849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.343866 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:25.343960 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.343965 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.345346 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.345350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.346468 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.346478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.347585 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.347590 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.347671 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347694 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.347760 (+ 66us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.347864 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260811 04:39:25.348464 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.342698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.2.238.131:37741)
0811 04:39:25.343455 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.343466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.343489 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:39:25.343555 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.343559 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.343712 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.344097 (+ 385us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:25.344249 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.345228 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.346586 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.347307 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.347416 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347509 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.348191 (+ 682us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.348313 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":154,"threads_started":1}
I20260811 04:39:25.348589 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.341759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:41465 (local address 127.0.0.1:48498)
0811 04:39:25.342679 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.342692 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.342790 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.343544 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.343549 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.343571 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:25.343710 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.343717 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.344735 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.344741 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.346081 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.346091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.347127 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.347130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.347346 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.347476 (+ 130us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.347580 (+ 104us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.348496 (+ 916us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":859,"thread_start_us":106,"threads_started":1}
I20260811 04:39:25.351166 20008 tablet_service.cc:1467] Processing CreateTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e (DEFAULT_TABLE table=test-table [id=69450c6282fa49b59b3d72515b7e3bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:25.351436 19718 tablet_service.cc:1467] Processing CreateTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e (DEFAULT_TABLE table=test-table [id=69450c6282fa49b59b3d72515b7e3bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:25.352298 20008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2eeb94f18a40339f3dd2f6d7a6215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:25.352507 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2eeb94f18a40339f3dd2f6d7a6215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:25.353711 19863 tablet_service.cc:1467] Processing CreateTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e (DEFAULT_TABLE table=test-table [id=69450c6282fa49b59b3d72515b7e3bca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:25.354750 19863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe2eeb94f18a40339f3dd2f6d7a6215e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:25.358806 20097 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Bootstrap starting.
I20260811 04:39:25.360405 20098 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Bootstrap starting.
I20260811 04:39:25.361668 20097 tablet_bootstrap.cc:654] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:25.361899 20099 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Bootstrap starting.
I20260811 04:39:25.362699 20098 tablet_bootstrap.cc:654] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:25.362710 20097 log.cc:824] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:25.363834 20098 log.cc:824] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:25.364315 20099 tablet_bootstrap.cc:654] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:25.365052 20097 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: No bootstrap required, opened a new log
I20260811 04:39:25.365105 20099 log.cc:824] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:25.365258 20097 ts_tablet_manager.cc:1397] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260811 04:39:25.365630 20098 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: No bootstrap required, opened a new log
I20260811 04:39:25.365856 20098 ts_tablet_manager.cc:1397] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:25.367093 20099 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: No bootstrap required, opened a new log
I20260811 04:39:25.367290 20099 ts_tablet_manager.cc:1397] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:39:25.368681 20098 raft_consensus.cc:348] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.368923 20098 raft_consensus.cc:374] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:25.368976 20098 raft_consensus.cc:729] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6464be8500143359c73bbb30fb7b704, State: Initialized, Role: FOLLOWER
I20260811 04:39:25.369416 20098 consensus_queue.cc:260] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.370086 19786 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
I20260811 04:39:25.369122 20097 raft_consensus.cc:348] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.370240 20097 raft_consensus.cc:374] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:25.370302 20097 raft_consensus.cc:729] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3bc6f24a0746dcabe1a67c6f22f408, State: Initialized, Role: FOLLOWER
I20260811 04:39:25.370427 20098 ts_tablet_manager.cc:1428] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260811 04:39:25.370790 20097 consensus_queue.cc:260] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.371022 20099 raft_consensus.cc:348] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.371297 20099 raft_consensus.cc:374] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:25.371357 20099 raft_consensus.cc:729] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b8db948e9045aaabeb9467f7031d67, State: Initialized, Role: FOLLOWER
I20260811 04:39:25.371811 20097 ts_tablet_manager.cc:1428] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260811 04:39:25.371783 20099 consensus_queue.cc:260] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.372481 19931 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
I20260811 04:39:25.372642 20099 ts_tablet_manager.cc:1428] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:39:25.373344 20077 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
W20260811 04:39:25.375674 19787 tablet.cc:2369] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:25.415545 20104 raft_consensus.cc:482] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:25.415977 20104 raft_consensus.cc:504] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.417204 20104 leader_election.cc:288] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585), 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465)
I20260811 04:39:25.422676 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.417647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:54751 (local address 127.2.238.130:41465)
0811 04:39:25.417853 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.417857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.417946 (+ 89us) server_negotiation.cc:407] Connection header received
0811 04:39:25.418106 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.418109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.418164 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.418249 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:25.418632 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.419453 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.421317 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.421948 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.422085 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.422137 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.422479 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.422548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260811 04:39:25.422677 20095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.417519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:40129 (local address 127.2.238.129:44585)
0811 04:39:25.417656 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.417671 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.417896 (+ 225us) server_negotiation.cc:407] Connection header received
0811 04:39:25.418121 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.418124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.418173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.418249 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:25.418632 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.419453 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.421318 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.421948 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.422085 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.422137 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.422479 (+ 342us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.422548 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:39:25.422720 20079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.417423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.2.238.131:40129)
0811 04:39:25.417786 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.417801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.417997 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.418277 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.418280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.418306 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:25.418492 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.418497 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.419604 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.419608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.421178 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.421190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.422115 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.422119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.422278 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.422366 (+ 88us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.422447 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.422543 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":249,"spinlock_wait_cycles":20864}
I20260811 04:39:25.422737 20106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.417452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:41465 (local address 127.2.238.131:54751)
0811 04:39:25.417866 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.417880 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:25.417997 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:25.418277 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:25.418280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:25.418306 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:25.418489 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.418497 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.419604 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.419608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.421178 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:25.421190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.422115 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.422119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:25.422306 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.422366 (+ 60us) client_negotiation.cc:769] Sending connection context
0811 04:39:25.422447 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 04:39:25.422543 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":248,"spinlock_wait_cycles":29312,"thread_start_us":124,"threads_started":1}
I20260811 04:39:25.423405 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2eeb94f18a40339f3dd2f6d7a6215e" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6464be8500143359c73bbb30fb7b704" is_pre_election: true
I20260811 04:39:25.423419 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2eeb94f18a40339f3dd2f6d7a6215e" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8db948e9045aaabeb9467f7031d67" is_pre_election: true
I20260811 04:39:25.423722 19883 raft_consensus.cc:2454] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 0.
I20260811 04:39:25.423713 19738 raft_consensus.cc:2454] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 0.
I20260811 04:39:25.424288 19959 leader_election.cc:302] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408, c6464be8500143359c73bbb30fb7b704; no voters:
I20260811 04:39:25.424595 20104 raft_consensus.cc:2781] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:39:25.424669 20104 raft_consensus.cc:482] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:39:25.424719 20104 raft_consensus.cc:3037] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:25.425760 20104 raft_consensus.cc:504] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.426170 20104 leader_election.cc:288] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 election: Requested vote from peers c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585), 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465)
I20260811 04:39:25.426563 19883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2eeb94f18a40339f3dd2f6d7a6215e" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8db948e9045aaabeb9467f7031d67"
I20260811 04:39:25.426565 19738 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe2eeb94f18a40339f3dd2f6d7a6215e" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6464be8500143359c73bbb30fb7b704"
I20260811 04:39:25.426668 19883 raft_consensus.cc:3037] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:25.426702 19738 raft_consensus.cc:3037] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:25.427759 19738 raft_consensus.cc:2454] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 1.
I20260811 04:39:25.427786 19883 raft_consensus.cc:2454] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 1.
I20260811 04:39:25.428097 19958 leader_election.cc:302] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3bc6f24a0746dcabe1a67c6f22f408, c6464be8500143359c73bbb30fb7b704; no voters:
I20260811 04:39:25.428321 20104 raft_consensus.cc:2781] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:25.428542 20104 raft_consensus.cc:686] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 LEADER]: Becoming Leader. State: Replica: ac3bc6f24a0746dcabe1a67c6f22f408, State: Running, Role: LEADER
I20260811 04:39:25.428821 20104 consensus_queue.cc:237] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:25.432207 19576 catalog_manager.cc:5495] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } health_report { overall_health: HEALTHY } } }
I20260811 04:39:25.445245 20096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.441532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54392 (local address 127.2.238.131:37741)
0811 04:39:25.441674 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:25.441679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:25.441693 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:39:25.441928 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:25.441930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:25.441987 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:25.442103 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:25.442356 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.442871 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.443735 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:25.443960 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:25.444074 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:25.444389 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:25.444839 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:39:25.444979 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:25.445043 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 04:39:25.445101 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:39:25.451236 19738 raft_consensus.cc:1261] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Refusing update from remote peer ac3bc6f24a0746dcabe1a67c6f22f408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:25.451233 19883 raft_consensus.cc:1261] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Refusing update from remote peer ac3bc6f24a0746dcabe1a67c6f22f408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:25.451980 20107 consensus_queue.cc:1035] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:25.452211 20104 consensus_queue.cc:1035] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:25.460369 20111 mvcc.cc:205] Tried to move back new op lower bound from 7317189285681127424 to 7317189285599068160. Current Snapshot: MvccSnapshot[applied={T|T < 7317189285681127424}]
I20260811 04:39:25.467680 19582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:25.467852 19582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:25.469976 19582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Sending DeleteTablet for 3 replicas of tablet fe2eeb94f18a40339f3dd2f6d7a6215e
I20260811 04:39:25.470767 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19648
I20260811 04:39:25.470706 19863 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:25 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:48498
I20260811 04:39:25.470791 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:25 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:38496
I20260811 04:39:25.472851 20122 tablet_replica.cc:330] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: stopping tablet replica
I20260811 04:39:25.473212 20122 raft_consensus.cc:2229] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:25.473585 20122 raft_consensus.cc:2258] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:25.475033 20122 ts_tablet_manager.cc:1905] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:25.477808 20122 ts_tablet_manager.cc:1918] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:25.477905 20122 log.cc:1196] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/fe2eeb94f18a40339f3dd2f6d7a6215e
I20260811 04:39:25.478255 20122 ts_tablet_manager.cc:1939] T fe2eeb94f18a40339f3dd2f6d7a6215e P 21b8db948e9045aaabeb9467f7031d67: Deleting consensus metadata
I20260811 04:39:25.478313 20008 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:25 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:54388
I20260811 04:39:25.478909 20123 tablet_replica.cc:330] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: stopping tablet replica
I20260811 04:39:25.479271 20123 raft_consensus.cc:2229] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:39:25.479419 19567 catalog_manager.cc:4841] TS 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): tablet fe2eeb94f18a40339f3dd2f6d7a6215e (table test-table [id=69450c6282fa49b59b3d72515b7e3bca]) successfully deleted
I20260811 04:39:25.479743 20123 raft_consensus.cc:2258] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260811 04:39:25.479977 19566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:44585 (error 108)
I20260811 04:39:25.480829 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19791
I20260811 04:39:25.482537 20123 ts_tablet_manager.cc:1905] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:25.483836 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:25.483460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.0.0.1:38508)
0811 04:39:25.483627 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:25.483719 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:44585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260811 04:39:25.484123 19566 catalog_manager.cc:4584] TS c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe2eeb94f18a40339f3dd2f6d7a6215e: Network error: Client connection negotiation failed: client connection to 127.2.238.129:44585: connect: Connection refused (error 111)
I20260811 04:39:25.485567 20123 ts_tablet_manager.cc:1918] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:25.485666 20123 log.cc:1196] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/fe2eeb94f18a40339f3dd2f6d7a6215e
I20260811 04:39:25.486049 20123 ts_tablet_manager.cc:1939] T fe2eeb94f18a40339f3dd2f6d7a6215e P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting consensus metadata
I20260811 04:39:25.487210 19566 catalog_manager.cc:4841] TS ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): tablet fe2eeb94f18a40339f3dd2f6d7a6215e (table test-table [id=69450c6282fa49b59b3d72515b7e3bca]) successfully deleted
I20260811 04:39:25.490937 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19938
I20260811 04:39:25.497908 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 19547
I20260811 04:39:45.507033 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--webserver_interface=127.2.238.190
--webserver_port=43257
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:39:45.619300 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:45.619590 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:45.619674 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:45.623795 20125 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 04:39:45.623847 20125 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 04:39:45.623863 20125 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 04:39:45.623878 20125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:39:45.623896 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:45.623914 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:45.623960 20125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:39:45.623993 20125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:39:45.629256 20125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:45.630352 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:45.631409 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:45.645720 20135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:45.645735 20125 server_base.cc:1034] running on GCE node
W20260811 04:39:45.645735 20133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:45.646003 20132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:45.646677 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:45.647884 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:45.649073 20125 hybrid_clock.cc:648] HybridClock initialized: now 1786423185649057 us; error 28 us; skew 500 ppm
Aug 11 04:39:45 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423185, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:45.655194 20125 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:39:45.656121 20125 webserver.cc:460] Webserver started at http://127.2.238.190:43257/ using document root <none> and password file <none>
I20260811 04:39:45.656752 20125 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:45.656859 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:45.660965 20125 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:39:45.663262 20142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:45.664422 20125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:39:45.664551 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "3c3857a3548647dd97d1f896caa8cc19"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "f18b593221a15ddc5f201b5d33dc95d8"
server_key_iv: "20be476ffb77776c11251eba2d5a3316"
server_key_version: "encryptionkey@0"
I20260811 04:39:45.665112 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:45.681233 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:45.684087 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:45.684290 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:45.691419 20125 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39749
I20260811 04:39:45.691721 20195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39749 every 8 connection(s)
I20260811 04:39:45.692543 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:39:45.694758 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20125
I20260811 04:39:45.695178 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:44585
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=44815
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:45.697018 20196 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:39:45.698601 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap starting.
I20260811 04:39:45.716519 20196 log.cc:824] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:45.725286 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:45.725737 20196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap complete.
I20260811 04:39:45.729991 20196 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:45.730425 20196 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Initialized, Role: FOLLOWER
I20260811 04:39:45.730911 20196 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:45.731056 20196 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:39:45.731146 20196 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:39:45.731271 20196 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 1 FOLLOWER]: Advancing to term 2
I20260811 04:39:45.733584 20196 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:45.733981 20196 leader_election.cc:302] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c3857a3548647dd97d1f896caa8cc19; no voters:
I20260811 04:39:45.734314 20196 leader_election.cc:288] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260811 04:39:45.734459 20200 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Leader election won for term 2
I20260811 04:39:45.734732 20200 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 LEADER]: Becoming Leader. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Running, Role: LEADER
I20260811 04:39:45.735113 20200 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:39:45.735579 20196 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:39:45.736104 20200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c3857a3548647dd97d1f896caa8cc19. Latest consensus state: current_term: 2 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:39:45.736217 20200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:45.736552 20204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:39:45.736675 20201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:39:45.736863 20201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:39:45.741858 20204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:39:45.742570 20204 catalog_manager.cc:1259] Loaded cluster ID: 2adcdef53c674128bc224b509c12594a
I20260811 04:39:45.742638 20204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:39:45.744985 20204 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:39:45.745675 20204 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Loaded TSK: 0
I20260811 04:39:45.757598 20204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Generated new TSK 1
I20260811 04:39:45.757850 20204 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:39:45.814491 20198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:45.814752 20198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:45.814796 20198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:45.818974 20198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:45.819046 20198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:45.819133 20198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:39:45.824613 20198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:44585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=44815
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:45.825697 20198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:45.826747 20198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:45.836191 20226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:45.836437 20198 server_base.cc:1034] running on GCE node
W20260811 04:39:45.836241 20225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:45.836688 20230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:45.837126 20198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:45.837920 20198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:45.839097 20198 hybrid_clock.cc:648] HybridClock initialized: now 1786423185839064 us; error 53 us; skew 500 ppm
Aug 11 04:39:45 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423185, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:45.842127 20198 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:39:45.843263 20198 webserver.cc:460] Webserver started at http://127.2.238.129:44815/ using document root <none> and password file <none>
I20260811 04:39:45.843999 20198 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:45.844116 20198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:45.849346 20198 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:39:45.852095 20237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:45.853567 20198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260811 04:39:45.853716 20198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "c6464be8500143359c73bbb30fb7b704"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "a87ade3e84145568387e9fc4930ef383"
server_key_iv: "2e672580f58633b0a65d5f3ecf4e4d20"
server_key_version: "encryptionkey@0"
I20260811 04:39:45.854485 20198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:45.876633 20198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:45.880544 20198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:45.880743 20198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:45.881446 20198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:45.882764 20245 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 04:39:45.887110 20198 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 04:39:45.887187 20198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260811 04:39:45.887255 20198 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 04:39:45.889473 20198 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 04:39:45.889590 20198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:45.890025 20245 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Bootstrap starting.
I20260811 04:39:45.903061 20198 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44585
I20260811 04:39:45.903115 20363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44585 every 8 connection(s)
I20260811 04:39:45.904433 20198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:39:45.911944 20245 log.cc:824] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:45.912634 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20198
I20260811 04:39:45.913093 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:41465
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=35295
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 11 04:39:45 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423185, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:45.921346 20245 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:39:45.921926 20245 tablet_bootstrap.cc:492] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Bootstrap complete.
I20260811 04:39:45.921954 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:45.906607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:57819 (local address 127.2.238.190:39749)
0811 04:39:45.907059 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:45.907085 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:45.907351 (+ 266us) server_negotiation.cc:407] Connection header received
0811 04:39:45.908498 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:45.908513 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:45.908922 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:45.909331 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:45.910393 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.911832 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.913279 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.913752 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.913942 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:45.916769 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:45.916773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:45.916786 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:45.916811 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:45.918583 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:45.919113 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:45.919119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:45.919123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:45.919239 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:45.919651 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:45.919656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:45.919661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:45.920041 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:45.920183 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:45.920733 (+ 550us) server_negotiation.cc:299] Negotiation successful
0811 04:39:45.921002 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":136,"threads_started":1}
I20260811 04:39:45.922103 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:45.906539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:57819)
0811 04:39:45.907161 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:45.907213 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:45.908003 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:45.909346 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:45.909352 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:45.909688 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:45.910222 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:45.910237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.911982 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.911984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:45.913119 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:45.913128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.913917 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.913920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:45.914100 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:45.914661 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:45.914677 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:45.916549 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:45.918705 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:45.918711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:45.918725 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:45.918981 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:45.919333 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:45.919338 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:45.919340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:45.919497 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:45.920170 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:45.920177 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:45.920423 (+ 246us) client_negotiation.cc:769] Sending connection context
0811 04:39:45.920605 (+ 182us) client_negotiation.cc:240] Negotiation successful
0811 04:39:45.920943 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":122,"threads_started":1}
I20260811 04:39:45.923570 20245 ts_tablet_manager.cc:1397] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Time spent bootstrapping tablet: real 0.034s user 0.028s sys 0.004s
I20260811 04:39:45.924427 20364 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:45.924666 20364 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:45.925149 20364 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
I20260811 04:39:45.927439 20160 ts_manager.cc:194] Registered new tserver with Master: c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585)
I20260811 04:39:45.927318 20245 raft_consensus.cc:348] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:45.927802 20245 raft_consensus.cc:729] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6464be8500143359c73bbb30fb7b704, State: Initialized, Role: FOLLOWER
I20260811 04:39:45.928282 20245 consensus_queue.cc:260] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } } peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } }
I20260811 04:39:45.929224 20245 ts_tablet_manager.cc:1428] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:39:45.930774 20160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:57819
I20260811 04:39:45.933354 20364 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
Aug 11 04:39:45 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423185, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.129@KRBTEST.COM
I20260811 04:39:45.941740 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:45.930361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.0.0.1:52418)
0811 04:39:45.930732 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:45.930772 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:45.930887 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:45.932347 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:45.932353 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:45.932779 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:45.932900 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:45.932907 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.934156 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.934162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:45.934923 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:45.934949 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.935753 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.935757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:45.935855 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:45.936518 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:45.936532 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:45.938026 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:45.940182 (+ 2156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:45.940190 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:45.940195 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:45.940415 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:45.940711 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:45.940721 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:45.940722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:45.940798 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:45.941264 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:45.941268 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:45.941362 (+ 94us) client_negotiation.cc:769] Sending connection context
0811 04:39:45.941469 (+ 107us) client_negotiation.cc:240] Negotiation successful
0811 04:39:45.941607 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":106,"threads_started":1}
I20260811 04:39:45.941890 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:45.930439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52418 (local address 127.2.238.129:44585)
0811 04:39:45.931588 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:45.931597 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:45.931615 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:39:45.931666 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:45.931669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:45.931872 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:45.932341 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:45.933020 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.934061 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.935046 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:45.935607 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:45.935706 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:45.938180 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:45.938190 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:45.938204 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:45.938229 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:45.940059 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:45.940520 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:45.940526 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:45.940528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:45.940620 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:45.940881 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:45.940884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:45.940886 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:45.941153 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:45.941283 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:45.941567 (+ 284us) server_negotiation.cc:299] Negotiation successful
0811 04:39:45.941710 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1056,"thread_start_us":81,"threads_started":1}
I20260811 04:39:45.942767 20286 tablet_service.cc:1514] Processing DeleteTablet for tablet fe2eeb94f18a40339f3dd2f6d7a6215e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:25 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52418
I20260811 04:39:45.943531 20374 tablet_replica.cc:330] stopping tablet replica
I20260811 04:39:45.943825 20374 raft_consensus.cc:2229] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:45.944089 20374 raft_consensus.cc:2258] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:45.945025 20374 ts_tablet_manager.cc:1905] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:45.948674 20374 ts_tablet_manager.cc:1918] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:45.948783 20374 log.cc:1196] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal/wals/fe2eeb94f18a40339f3dd2f6d7a6215e
I20260811 04:39:45.949141 20374 ts_tablet_manager.cc:1939] T fe2eeb94f18a40339f3dd2f6d7a6215e P c6464be8500143359c73bbb30fb7b704: Deleting consensus metadata
I20260811 04:39:45.950934 20145 catalog_manager.cc:4841] TS c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585): tablet fe2eeb94f18a40339f3dd2f6d7a6215e (table test-table [id=69450c6282fa49b59b3d72515b7e3bca]) successfully deleted
W20260811 04:39:46.035439 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:46.035697 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:46.035739 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:46.039767 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:46.039853 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:46.039963 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:39:46.045236 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:41465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=35295
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:46.046276 20369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:46.047154 20369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:46.056128 20382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:46.056144 20380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:46.056131 20379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:46.056303 20369 server_base.cc:1034] running on GCE node
I20260811 04:39:46.056743 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:46.057337 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:46.058485 20369 hybrid_clock.cc:648] HybridClock initialized: now 1786423186058448 us; error 48 us; skew 500 ppm
Aug 11 04:39:46 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423186, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:46.061038 20369 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:39:46.061925 20369 webserver.cc:460] Webserver started at http://127.2.238.130:35295/ using document root <none> and password file <none>
I20260811 04:39:46.062521 20369 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:46.062625 20369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:46.066133 20369 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260811 04:39:46.067965 20390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.069118 20369 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:46.069268 20369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "21b8db948e9045aaabeb9467f7031d67"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "456a95b8fd6c8804306b937e479c0790"
server_key_iv: "be558f930a15788c4b9509cf02c607fe"
server_key_version: "encryptionkey@0"
I20260811 04:39:46.069778 20369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:46.088625 20369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:46.091516 20369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:46.091718 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:46.092327 20369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:46.093143 20369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:46.093207 20369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.093290 20369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:46.093333 20369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.101814 20369 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:41465
I20260811 04:39:46.101836 20511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:41465 every 8 connection(s)
I20260811 04:39:46.102888 20369 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
Aug 11 04:39:46 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423186, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:46.111774 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20369
I20260811 04:39:46.112258 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:37741
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=43523
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:39:46.115938 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.104655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53681 (local address 127.2.238.190:39749)
0811 04:39:46.104798 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.104802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.105856 (+ 1054us) server_negotiation.cc:407] Connection header received
0811 04:39:46.106630 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.106632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.106683 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.106766 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:46.107898 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.108374 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.109347 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.109550 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.109622 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.112097 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:46.112100 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:46.112102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:46.112125 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:46.113633 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.114268 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.114273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.114290 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.114387 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.114704 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.114708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.114711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.114982 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:46.115083 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.115544 (+ 461us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.115689 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:39:46.116482 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.105039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:53681)
0811 04:39:46.105695 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.105735 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.106460 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.106926 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.106932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.107291 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:46.107759 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.107773 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.108469 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.108471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.109250 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.109259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.109674 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.109676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.109829 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.110383 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:46.110416 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:46.111931 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:46.113749 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:46.113758 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:46.113776 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:46.114162 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:46.114480 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:46.114482 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:46.114483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:46.114603 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:46.115075 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:46.115079 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:46.115275 (+ 196us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.115428 (+ 153us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.115668 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":139,"threads_started":1}
I20260811 04:39:46.117659 20513 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:46.117902 20513 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:46.118356 20513 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
I20260811 04:39:46.119355 20160 ts_manager.cc:194] Registered new tserver with Master: 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465)
I20260811 04:39:46.119971 20160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:53681
W20260811 04:39:46.225076 20516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:39:46.225340 20516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:39:46.225378 20516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:39:46.229199 20516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:39:46.229255 20516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:39:46.229341 20516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:39:46.234509 20516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=43523
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:39:46.235467 20516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:39:46.236493 20516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:39:46.246464 20525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:46.246521 20516 server_base.cc:1034] running on GCE node
W20260811 04:39:46.246450 20521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:39:46.246438 20523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:39:46.247076 20516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:39:46.247641 20516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:39:46.248806 20516 hybrid_clock.cc:648] HybridClock initialized: now 1786423186248781 us; error 23 us; skew 500 ppm
Aug 11 04:39:46 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423186, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:39:46.251458 20516 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:39:46.252321 20516 webserver.cc:460] Webserver started at http://127.2.238.131:43523/ using document root <none> and password file <none>
I20260811 04:39:46.252980 20516 fs_manager.cc:359] Metadata directory not provided
I20260811 04:39:46.253054 20516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:39:46.256539 20516 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:39:46.258978 20533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.260231 20516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:39:46.260354 20516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "ac3bc6f24a0746dcabe1a67c6f22f408"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "c7336f2ed7a43a4cdb69f194ffac234a"
server_key_iv: "1638d3150fb55ed434badc8e83a57137"
server_key_version: "encryptionkey@0"
I20260811 04:39:46.260890 20516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:39:46.299489 20516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:39:46.303119 20516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:39:46.303352 20516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:39:46.304018 20516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:39:46.304906 20516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:39:46.304973 20516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.305040 20516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:39:46.305083 20516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:39:46.314320 20516 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:37741
I20260811 04:39:46.314345 20651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:37741 every 8 connection(s)
I20260811 04:39:46.315375 20516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:39:46.320555 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20516
Aug 11 04:39:46 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423186, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:39:46.328974 20656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.321245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53150 (local address 127.2.238.190:39749)
0811 04:39:46.321527 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.321530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.321638 (+ 108us) server_negotiation.cc:407] Connection header received
0811 04:39:46.321758 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.321760 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.321847 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.321924 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:46.322861 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.323350 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.324190 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.324520 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.324622 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.325976 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:46.325979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:46.325982 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:46.326005 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:46.327400 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.327847 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.327853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.327857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.327931 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.328182 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.328188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.328190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.328384 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:46.328576 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.328701 (+ 125us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.328792 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":115,"threads_started":1}
I20260811 04:39:46.329843 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.317166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:51819 (local address 127.2.238.190:39749)
0811 04:39:46.317305 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.317309 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.318689 (+ 1380us) server_negotiation.cc:407] Connection header received
0811 04:39:46.319553 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.319556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.319621 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.319730 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:46.320983 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.321730 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.323301 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.323516 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.323614 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.326115 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:46.326118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:46.326120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:46.326156 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:46.327668 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.328236 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.328238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.328239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.328307 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.328651 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.328654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.328655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.328876 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:46.328969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.329620 (+ 651us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.329707 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:39:46.330960 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.317500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:51819)
0811 04:39:46.318524 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.318587 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.319338 (+ 751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.319894 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.319900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.320262 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:39:46.320817 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.320833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.321881 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.321889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.323180 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.323189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.323652 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.323656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.323811 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.324374 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:39:46.324393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:39:46.325961 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:39:46.327771 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:46.327775 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:46.327786 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:46.328126 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:46.328389 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:39:46.328392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:39:46.328393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:39:46.328551 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:39:46.329028 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:39:46.329033 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:39:46.329358 (+ 325us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.329567 (+ 209us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.329913 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":668,"threads_started":1}
I20260811 04:39:46.331939 20652 heartbeater.cc:347] Connected to a master server at 127.2.238.190:39749
I20260811 04:39:46.332177 20652 heartbeater.cc:464] Registering TS with master...
I20260811 04:39:46.332667 20652 heartbeater.cc:511] Master 127.2.238.190:39749 requested a full tablet report, sending...
I20260811 04:39:46.333616 20160 ts_manager.cc:194] Registered new tserver with Master: ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
I20260811 04:39:46.334208 20160 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:51819
I20260811 04:39:46.342661 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:39:46.346082 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.343272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.2.238.190:39749)
0811 04:39:46.343371 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.343374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.343558 (+ 184us) server_negotiation.cc:407] Connection header received
0811 04:39:46.343683 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.343685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.343771 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.343847 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:46.344033 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.344475 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.345170 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.345341 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.345404 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.345643 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:46.345758 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0811 04:39:46.345981 (+ 223us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53166: token expired
Metrics: {"server-negotiator.queue_time_us":37}
W20260811 04:39:46.346136 20366 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53166: token expired
W20260811 04:39:46.346522 20081 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.238.190:39749)
I20260811 04:39:46.353087 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.346814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.2.238.190:39749)
0811 04:39:46.346920 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.346922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.346962 (+ 40us) server_negotiation.cc:407] Connection header received
0811 04:39:46.347058 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.347060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.347089 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.347155 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:39:46.347652 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.348046 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.348752 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.348948 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.349025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.350315 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:39:46.350317 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:39:46.350319 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:39:46.350339 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:39:46.351553 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.351972 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.351975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.351977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.352054 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:39:46.352326 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:39:46.352330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:39:46.352332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:39:46.352491 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:39:46.352629 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.352845 (+ 216us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.352950 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:39:46.355438 20160 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:39:46.358847 20160 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:39:46.367041 20286 tablet_service.cc:1467] Processing CreateTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf (DEFAULT_TABLE table=test-table [id=475356aed7c54ceb902ed26b27c31651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:46.367770 20286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d3cf58e1fc4d2db7f7b217d880afdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:46.370873 20245 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Bootstrap starting.
I20260811 04:39:46.371750 20245 tablet_bootstrap.cc:654] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:46.372346 20371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.365755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:41465 (local address 127.0.0.1:43238)
0811 04:39:46.365956 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.365974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.366053 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.367447 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.367450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.367480 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:46.367756 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.367764 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.369389 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.369394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.370631 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.370641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.371797 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.371801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.371914 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.371973 (+ 59us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.372068 (+ 95us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.372176 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":65,"spinlock_wait_cycles":6272}
I20260811 04:39:46.372545 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.366049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:37741 (local address 127.0.0.1:40574)
0811 04:39:46.366421 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.366435 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.366526 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.367447 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.367450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.367481 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:46.368114 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.368122 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.369470 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.369472 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.370828 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.370848 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.372079 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.372083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.372174 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.372196 (+ 22us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.372270 (+ 74us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.372342 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":680,"spinlock_wait_cycles":8960,"thread_start_us":106,"threads_started":1}
I20260811 04:39:46.372954 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.366244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.2.238.130:41465)
0811 04:39:46.366945 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.366953 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.366972 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:39:46.367037 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.367041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.367243 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.367897 (+ 654us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:46.368110 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.369266 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.370793 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.371658 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.371774 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.371932 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.372651 (+ 719us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.372799 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":113,"threads_started":1}
I20260811 04:39:46.373198 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.366595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.2.238.131:37741)
0811 04:39:46.366928 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.366941 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.366965 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:39:46.367040 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.367045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.367251 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.367451 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:46.368250 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.369368 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.371053 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.371947 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.372086 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.372228 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.372298 (+ 70us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.372402 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
I20260811 04:39:46.374513 20245 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: No bootstrap required, opened a new log
I20260811 04:39:46.374629 20245 ts_tablet_manager.cc:1397] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260811 04:39:46.375123 20245 raft_consensus.cc:348] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.375254 20245 raft_consensus.cc:374] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:46.375305 20245 raft_consensus.cc:729] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6464be8500143359c73bbb30fb7b704, State: Initialized, Role: FOLLOWER
I20260811 04:39:46.375470 20245 consensus_queue.cc:260] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.375784 20245 ts_tablet_manager.cc:1428] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260811 04:39:46.376230 20583 tablet_service.cc:1467] Processing CreateTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf (DEFAULT_TABLE table=test-table [id=475356aed7c54ceb902ed26b27c31651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:46.376194 20442 tablet_service.cc:1467] Processing CreateTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf (DEFAULT_TABLE table=test-table [id=475356aed7c54ceb902ed26b27c31651]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:39:46.377233 20583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d3cf58e1fc4d2db7f7b217d880afdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:46.377333 20442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79d3cf58e1fc4d2db7f7b217d880afdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:39:46.384330 20666 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Bootstrap starting.
I20260811 04:39:46.384681 20667 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Bootstrap starting.
I20260811 04:39:46.387180 20666 tablet_bootstrap.cc:654] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:46.387535 20667 tablet_bootstrap.cc:654] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Neither blocks nor log segments found. Creating new log.
I20260811 04:39:46.387995 20666 log.cc:824] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:46.388236 20667 log.cc:824] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Log is configured to *not* fsync() on all Append() calls
I20260811 04:39:46.390323 20667 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: No bootstrap required, opened a new log
I20260811 04:39:46.390323 20666 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: No bootstrap required, opened a new log
I20260811 04:39:46.390528 20666 ts_tablet_manager.cc:1397] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:46.390528 20667 ts_tablet_manager.cc:1397] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:39:46.394708 20666 raft_consensus.cc:348] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.394708 20667 raft_consensus.cc:348] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.395020 20666 raft_consensus.cc:374] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:46.395020 20667 raft_consensus.cc:374] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:39:46.395089 20666 raft_consensus.cc:729] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac3bc6f24a0746dcabe1a67c6f22f408, State: Initialized, Role: FOLLOWER
I20260811 04:39:46.395100 20667 raft_consensus.cc:729] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21b8db948e9045aaabeb9467f7031d67, State: Initialized, Role: FOLLOWER
I20260811 04:39:46.395591 20667 consensus_queue.cc:260] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.395584 20666 consensus_queue.cc:260] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.396307 20652 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
I20260811 04:39:46.396659 20666 ts_tablet_manager.cc:1428] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260811 04:39:46.396659 20667 ts_tablet_manager.cc:1428] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:39:46.397076 20513 heartbeater.cc:503] Master 127.2.238.190:39749 was elected leader, sending a full tablet report...
W20260811 04:39:46.405887 20365 tablet.cc:2369] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:39:46.406759 20672 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:39:46.406916 20672 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.407966 20672 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585)
I20260811 04:39:46.411674 20372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.408294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:53455 (local address 127.2.238.129:44585)
0811 04:39:46.408422 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.408427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.408441 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:39:46.408494 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.408496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.408546 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.408630 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:46.408897 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.409653 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.410688 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.411149 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.411275 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.411438 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.411493 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.411556 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:39:46.411665 20654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.408174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.2.238.131:53455)
0811 04:39:46.408312 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.408329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.408407 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.408642 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.408644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.408657 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:46.408762 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.408766 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.409755 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.409760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.410557 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.410567 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.411270 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.411273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.411386 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.411407 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.411468 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.411538 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 04:39:46.412343 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d3cf58e1fc4d2db7f7b217d880afdf" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6464be8500143359c73bbb30fb7b704" is_pre_election: true
I20260811 04:39:46.412524 20673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.408174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:41465 (local address 127.2.238.131:43555)
0811 04:39:46.409083 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.409094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:39:46.409193 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:39:46.409521 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:39:46.409523 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:39:46.409541 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:39:46.409666 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.409671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.410413 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.410419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.411160 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:39:46.411170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.412029 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.412034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:39:46.412145 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.412181 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 04:39:46.412294 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 04:39:46.412361 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":830,"mutex_wait_us":516,"thread_start_us":188,"threads_started":1}
I20260811 04:39:46.412540 20663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.408271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:43555 (local address 127.2.238.130:41465)
0811 04:39:46.408398 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.408403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.409190 (+ 787us) server_negotiation.cc:407] Connection header received
0811 04:39:46.409258 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.409261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.409307 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.409390 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:39:46.409825 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.410297 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.411285 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.411767 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.411887 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.411948 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.412335 (+ 387us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.412409 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:39:46.412716 20314 raft_consensus.cc:2454] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 0.
I20260811 04:39:46.413220 20536 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac3bc6f24a0746dcabe1a67c6f22f408, c6464be8500143359c73bbb30fb7b704; no voters:
I20260811 04:39:46.413267 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d3cf58e1fc4d2db7f7b217d880afdf" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8db948e9045aaabeb9467f7031d67" is_pre_election: true
I20260811 04:39:46.413511 20672 raft_consensus.cc:2781] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:39:46.413523 20464 raft_consensus.cc:2454] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 0.
I20260811 04:39:46.413602 20672 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:39:46.413667 20672 raft_consensus.cc:3037] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:46.414760 20672 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.415095 20672 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 election: Requested vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585)
I20260811 04:39:46.415443 20464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d3cf58e1fc4d2db7f7b217d880afdf" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21b8db948e9045aaabeb9467f7031d67"
I20260811 04:39:46.415443 20314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d3cf58e1fc4d2db7f7b217d880afdf" candidate_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6464be8500143359c73bbb30fb7b704"
I20260811 04:39:46.415561 20314 raft_consensus.cc:3037] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:46.415572 20464 raft_consensus.cc:3037] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:39:46.416862 20464 raft_consensus.cc:2454] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 1.
I20260811 04:39:46.416862 20314 raft_consensus.cc:2454] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac3bc6f24a0746dcabe1a67c6f22f408 in term 1.
I20260811 04:39:46.417172 20536 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408, c6464be8500143359c73bbb30fb7b704; no voters:
I20260811 04:39:46.417321 20672 raft_consensus.cc:2781] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:39:46.417552 20672 raft_consensus.cc:686] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 LEADER]: Becoming Leader. State: Replica: ac3bc6f24a0746dcabe1a67c6f22f408, State: Running, Role: LEADER
I20260811 04:39:46.417924 20672 consensus_queue.cc:237] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:39:46.421263 20160 catalog_manager.cc:5495] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } health_report { overall_health: UNKNOWN } } }
I20260811 04:39:46.444648 20664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:39:46.440887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40580 (local address 127.2.238.131:37741)
0811 04:39:46.441041 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:39:46.441045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:39:46.441063 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:39:46.441156 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:39:46.441159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:39:46.441224 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:39:46.441351 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:39:46.441592 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.442186 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.443138 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:39:46.443382 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:39:46.443491 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:39:46.443791 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:39:46.444248 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:39:46.444376 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0811 04:39:46.444436 (+ 60us) server_negotiation.cc:299] Negotiation successful
0811 04:39:46.444493 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 04:39:46.450289 20464 raft_consensus.cc:1261] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Refusing update from remote peer ac3bc6f24a0746dcabe1a67c6f22f408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:46.450285 20314 raft_consensus.cc:1261] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Refusing update from remote peer ac3bc6f24a0746dcabe1a67c6f22f408: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:39:46.450970 20672 consensus_queue.cc:1035] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:46.451166 20674 consensus_queue.cc:1035] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:39:46.457845 20677 mvcc.cc:205] Tried to move back new op lower bound from 7317189371693207552 to 7317189371570823168. Current Snapshot: MvccSnapshot[applied={T|T < 7317189371693207552}]
I20260811 04:39:46.459088 20678 mvcc.cc:205] Tried to move back new op lower bound from 7317189371693207552 to 7317189371570823168. Current Snapshot: MvccSnapshot[applied={T|T < 7317189371693207552}]
I20260811 04:39:46.465327 20158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:46.465512 20158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53182:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:39:46.467854 20158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Sending DeleteTablet for 3 replicas of tablet 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:39:46.468531 20286 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:46 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52418
I20260811 04:39:46.468544 20442 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:46 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:43238
I20260811 04:39:46.468729 20583 tablet_service.cc:1514] Processing DeleteTablet for tablet 79d3cf58e1fc4d2db7f7b217d880afdf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:39:46 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:40574
I20260811 04:39:46.469055 20687 tablet_replica.cc:330] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: stopping tablet replica
I20260811 04:39:46.469146 20688 tablet_replica.cc:330] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: stopping tablet replica
I20260811 04:39:46.469363 20687 raft_consensus.cc:2229] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:39:46.469385 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20198
I20260811 04:39:46.469460 20688 raft_consensus.cc:2229] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:39:46.469641 20687 raft_consensus.cc:2258] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:46.469811 20688 raft_consensus.cc:2258] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:39:46.472003 20688 ts_tablet_manager.cc:1905] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:46.476140 20688 ts_tablet_manager.cc:1918] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:46.476248 20688 log.cc:1196] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:39:46.476593 20688 ts_tablet_manager.cc:1939] T 79d3cf58e1fc4d2db7f7b217d880afdf P ac3bc6f24a0746dcabe1a67c6f22f408: Deleting consensus metadata
I20260811 04:39:46.477821 20145 catalog_manager.cc:4841] TS ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): tablet 79d3cf58e1fc4d2db7f7b217d880afdf (table test-table [id=475356aed7c54ceb902ed26b27c31651]) successfully deleted
I20260811 04:39:46.482935 20687 ts_tablet_manager.cc:1905] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:39:46.485030 20687 ts_tablet_manager.cc:1918] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:39:46.485132 20687 log.cc:1196] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:39:46.485476 20687 ts_tablet_manager.cc:1939] T 79d3cf58e1fc4d2db7f7b217d880afdf P 21b8db948e9045aaabeb9467f7031d67: Deleting consensus metadata
W20260811 04:39:46.485466 20145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:44585 (error 108)
I20260811 04:39:46.486708 20146 catalog_manager.cc:4841] TS 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): tablet 79d3cf58e1fc4d2db7f7b217d880afdf (table test-table [id=475356aed7c54ceb902ed26b27c31651]) successfully deleted
W20260811 04:39:46.486816 20662 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:39:46.486453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:44585 (local address 127.0.0.1:52430)
0811 04:39:46.486616 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 04:39:46.486693 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:44585: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 04:39:46.487071 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20369
W20260811 04:39:46.486945 20145 catalog_manager.cc:4584] TS c6464be8500143359c73bbb30fb7b704 (127.2.238.129:44585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79d3cf58e1fc4d2db7f7b217d880afdf: Network error: Client connection negotiation failed: client connection to 127.2.238.129:44585: connect: Connection reset by peer (error 104)
I20260811 04:39:46.496102 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20516
I20260811 04:39:46.503595 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20125
I20260811 04:40:06.511857 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--webserver_interface=127.2.238.190
--webserver_port=43257
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:06.621606 20690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:06.621929 20690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:06.622004 20690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:06.625801 20690 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260811 04:40:06.625875 20690 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260811 04:40:06.625898 20690 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260811 04:40:06.625919 20690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:06.625957 20690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:06.625993 20690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:06.626025 20690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:06.626057 20690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:06.630702 20690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=43257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:06.631669 20690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:06.632638 20690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:06.640865 20696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:06.640857 20698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:06.640941 20690 server_base.cc:1034] running on GCE node
W20260811 04:40:06.640858 20695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:06.641688 20690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:06.642726 20690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:06.643930 20690 hybrid_clock.cc:648] HybridClock initialized: now 1786423206643910 us; error 23 us; skew 500 ppm
Aug 11 04:40:06 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:06.652832 20690 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:06.653684 20690 webserver.cc:460] Webserver started at http://127.2.238.190:43257/ using document root <none> and password file <none>
I20260811 04:40:06.654268 20690 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:06.654353 20690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:06.658031 20690 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260811 04:40:06.660073 20705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:06.661257 20690 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:06.661401 20690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "3c3857a3548647dd97d1f896caa8cc19"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "f18b593221a15ddc5f201b5d33dc95d8"
server_key_iv: "20be476ffb77776c11251eba2d5a3316"
server_key_version: "encryptionkey@0"
I20260811 04:40:06.661995 20690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:06.691279 20690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:06.694434 20690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:06.694660 20690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:06.702001 20690 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39749
I20260811 04:40:06.702322 20760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39749 every 8 connection(s)
I20260811 04:40:06.703109 20690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:06.707417 20761 sys_catalog.cc:263] Verifying existing consensus state
I20260811 04:40:06.708992 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap starting.
I20260811 04:40:06.709363 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20690
I20260811 04:40:06.709839 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:44585
--local_ip_for_outbound_sockets=127.2.238.129
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=44815
--webserver_interface=127.2.238.129
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:40:06.727528 20761 log.cc:824] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:06.738997 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:40:06.739400 20761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Bootstrap complete.
I20260811 04:40:06.743577 20761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:40:06.743992 20761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Initialized, Role: FOLLOWER
I20260811 04:40:06.744526 20761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:40:06.744678 20761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:06.744769 20761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:06.744885 20761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 2 FOLLOWER]: Advancing to term 3
I20260811 04:40:06.746790 20761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:40:06.747136 20761 leader_election.cc:302] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c3857a3548647dd97d1f896caa8cc19; no voters:
I20260811 04:40:06.747494 20761 leader_election.cc:288] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260811 04:40:06.747601 20767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 3 FOLLOWER]: Leader election won for term 3
I20260811 04:40:06.747822 20767 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [term 3 LEADER]: Becoming Leader. State: Replica: 3c3857a3548647dd97d1f896caa8cc19, State: Running, Role: LEADER
I20260811 04:40:06.748106 20767 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } }
I20260811 04:40:06.748961 20761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:06.749579 20768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:40:06.749706 20768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:06.750013 20769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c3857a3548647dd97d1f896caa8cc19. Latest consensus state: current_term: 3 leader_uuid: "3c3857a3548647dd97d1f896caa8cc19" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c3857a3548647dd97d1f896caa8cc19" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39749 } } }
I20260811 04:40:06.750173 20769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:06.750473 20773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:06.755949 20773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:06.756949 20773 catalog_manager.cc:1259] Loaded cluster ID: 2adcdef53c674128bc224b509c12594a
I20260811 04:40:06.757030 20773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:06.759979 20773 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:06.760769 20773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Loaded TSK: 0
I20260811 04:40:06.760860 20773 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3c3857a3548647dd97d1f896caa8cc19: Loaded TSK: 1
F20260811 04:40:06.761659 20773 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8bd421f9ad google::LogMessage::Fail() at ??:0
@ 0x7f8bd422288a google::LogMessage::SendToLog() at ??:0
@ 0x7f8bd421f4de google::LogMessage::Flush() at ??:0
@ 0x7f8bd4220a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8bd6ee318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8bd6edbf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8bdeb847bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8bdeb5e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8bdeb5f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8bdeba0df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8bdebbf82b std::function<>::operator()() at ??:0
@ 0x7f8bdeb5eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8bdeb5ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8bdeb5c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8bdeba0640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8bd64713d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8bd6471c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8bd64733f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8bd6462512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8bd7aa66db start_thread at ??:0
@ 0x7f8bd2d1d61f clone at ??:0
W20260811 04:40:06.863111 20764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:06.863345 20764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:06.863391 20764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:06.867388 20764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:06.867439 20764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:06.867524 20764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:06.872591 20764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:44585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=44815
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:06.873672 20764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:06.874717 20764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:06.885458 20813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:06.885488 20814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:06.885713 20764 server_base.cc:1034] running on GCE node
W20260811 04:40:06.885878 20816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:06.886312 20764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:06.887080 20764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:06.888298 20764 hybrid_clock.cc:648] HybridClock initialized: now 1786423206888253 us; error 51 us; skew 500 ppm
Aug 11 04:40:06 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:06.891656 20764 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:40:06.892578 20764 webserver.cc:460] Webserver started at http://127.2.238.129:44815/ using document root <none> and password file <none>
I20260811 04:40:06.893151 20764 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:06.893214 20764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:06.897058 20764 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:06.899413 20823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:06.900755 20764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260811 04:40:06.900899 20764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "c6464be8500143359c73bbb30fb7b704"
format_stamp: "Formatted at 2026-08-11 04:39:24 on dist-test-slave-0kls"
server_key: "a87ade3e84145568387e9fc4930ef383"
server_key_iv: "2e672580f58633b0a65d5f3ecf4e4d20"
server_key_version: "encryptionkey@0"
I20260811 04:40:06.901443 20764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:06.914837 20764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:06.917827 20764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:06.918028 20764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:06.918664 20764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:06.919925 20830 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260811 04:40:06.925655 20764 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260811 04:40:06.925745 20764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260811 04:40:06.925846 20764 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260811 04:40:06.927431 20764 ts_tablet_manager.cc:610] Registered 1 tablets
I20260811 04:40:06.927497 20764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260811 04:40:06.927878 20830 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Bootstrap starting.
I20260811 04:40:06.937712 20764 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:44585
I20260811 04:40:06.938064 20943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:44585 every 8 connection(s)
I20260811 04:40:06.939031 20764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:06.948110 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20764
I20260811 04:40:06.948648 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:41465
--local_ip_for_outbound_sockets=127.2.238.130
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=35295
--webserver_interface=127.2.238.130
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:40:06.949442 20830 log.cc:824] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:06.958575 20830 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260811 04:40:06.959010 20830 tablet_bootstrap.cc:492] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Bootstrap complete.
I20260811 04:40:06.960544 20830 ts_tablet_manager.cc:1397] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20260811 04:40:06.963896 20830 raft_consensus.cc:348] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:06.964403 20830 raft_consensus.cc:729] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6464be8500143359c73bbb30fb7b704, State: Initialized, Role: FOLLOWER
I20260811 04:40:06.964943 20830 consensus_queue.cc:260] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:06.965919 20830 ts_tablet_manager.cc:1428] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260811 04:40:06.989481 20947 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:40:06.941040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:58915)
0811 04:40:06.941513 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:06.941564 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:06.942371 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:06.988770 (+ 46399us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":137,"threads_started":1}
W20260811 04:40:06.990113 20945 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39749 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39749: Client connection negotiation failed: client connection to 127.2.238.190:39749: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260811 04:40:06.990917 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:06.990581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:60033)
0811 04:40:06.990766 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:06.990863 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 04:40:06.991600 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:06.991387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:35397)
0811 04:40:06.991489 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:06.991534 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260811 04:40:06.991793 20945 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260811 04:40:07.067720 20949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:07.067962 20949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:07.068030 20949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:07.071946 20949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:07.072002 20949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:07.072057 20949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:07.077242 20949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:41465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=35295
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:07.078364 20949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:07.079432 20949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:07.089612 20958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:07.089689 20956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:07.089624 20955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:07.090018 20949 server_base.cc:1034] running on GCE node
I20260811 04:40:07.090499 20949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:07.091212 20949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:07.092410 20949 hybrid_clock.cc:648] HybridClock initialized: now 1786423207092395 us; error 26 us; skew 500 ppm
Aug 11 04:40:07 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:07.095316 20949 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:40:07.096189 20949 webserver.cc:460] Webserver started at http://127.2.238.130:35295/ using document root <none> and password file <none>
I20260811 04:40:07.096699 20949 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:07.096760 20949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:07.100561 20949 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:07.102596 20965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.103801 20949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:07.103935 20949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "21b8db948e9045aaabeb9467f7031d67"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "456a95b8fd6c8804306b937e479c0790"
server_key_iv: "be558f930a15788c4b9509cf02c607fe"
server_key_version: "encryptionkey@0"
I20260811 04:40:07.104486 20949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:07.115219 20949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:07.118323 20949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:07.118513 20949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:07.119194 20949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:07.120103 20949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:07.120163 20949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.120226 20949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:07.120256 20949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.129439 20949 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:41465
I20260811 04:40:07.129477 21086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:41465 every 8 connection(s)
I20260811 04:40:07.130502 20949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:07.134049 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.132495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:43997)
0811 04:40:07.133179 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.133281 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":105,"threads_started":1}
W20260811 04:40:07.134521 21087 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39749 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39749: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
I20260811 04:40:07.135047 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.134841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:33569)
0811 04:40:07.134955 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.134998 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260811 04:40:07.135620 21091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.135445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:48623)
0811 04:40:07.135535 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.135574 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 04:40:07.135777 21087 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 04:40:07.136735 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 20949
I20260811 04:40:07.137094 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:37741
--local_ip_for_outbound_sockets=127.2.238.131
--tserver_master_addrs=127.2.238.190:39749
--webserver_port=43523
--webserver_interface=127.2.238.131
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:32975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:07.247475 21092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:07.247759 21092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:07.247802 21092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:07.251719 21092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:07.251771 21092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:07.251827 21092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:07.256824 21092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:32975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:37741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=43523
--tserver_master_addrs=127.2.238.190:39749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:07.257776 21092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:07.258765 21092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:07.268649 21101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:07.268710 21097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:07.268724 21099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:07.268926 21092 server_base.cc:1034] running on GCE node
I20260811 04:40:07.269467 21092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:07.270305 21092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:07.271476 21092 hybrid_clock.cc:648] HybridClock initialized: now 1786423207271438 us; error 47 us; skew 500 ppm
Aug 11 04:40:07 dist-test-slave-0kls krb5kdc[19532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:07.274389 21092 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:40:07.275395 21092 webserver.cc:460] Webserver started at http://127.2.238.131:43523/ using document root <none> and password file <none>
I20260811 04:40:07.275979 21092 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:07.276075 21092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:07.280553 21092 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:40:07.282754 21108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.284040 21092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:07.284185 21092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "ac3bc6f24a0746dcabe1a67c6f22f408"
format_stamp: "Formatted at 2026-08-11 04:39:25 on dist-test-slave-0kls"
server_key: "c7336f2ed7a43a4cdb69f194ffac234a"
server_key_iv: "1638d3150fb55ed434badc8e83a57137"
server_key_version: "encryptionkey@0"
I20260811 04:40:07.284721 21092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:07.306216 21092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:07.309653 21092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:07.309899 21092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:07.310509 21092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:07.311425 21092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:07.311476 21092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.311520 21092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:07.311538 21092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:07.321753 21092 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:37741
I20260811 04:40:07.321835 21232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:37741 every 8 connection(s)
I20260811 04:40:07.322923 21092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:07.324311 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21092
I20260811 04:40:07.327000 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.325358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:44161)
0811 04:40:07.326167 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.326279 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":169,"threads_started":1}
W20260811 04:40:07.327461 21233 heartbeater.cc:644] Failed to heartbeat to 127.2.238.190:39749 (0 consecutive failures): Network error: Failed to ping master at 127.2.238.190:39749: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
I20260811 04:40:07.328022 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.327855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:35459)
0811 04:40:07.327930 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.327971 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260811 04:40:07.328542 21236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.328393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:49791)
0811 04:40:07.328456 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.328492 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260811 04:40:07.328733 21233 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260811 04:40:07.993034 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:07.992527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:60997)
0811 04:40:07.992842 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:07.992911 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":87,"threads_started":1}
I20260811 04:40:08.137095 21238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.136518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:45877)
0811 04:40:08.136949 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:08.137000 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":148,"threads_started":1}
I20260811 04:40:08.206400 21239 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:08.206593 21239 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:08.207916 21239 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
Aug 11 04:40:08 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.130@KRBTEST.COM
Aug 11 04:40:08 dist-test-slave-0kls krb5kdc[19532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.131@KRBTEST.COM
I20260811 04:40:08.225188 21237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.208073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:41465 (local address 127.2.238.129:47513)
0811 04:40:08.208459 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:08.208476 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:08.208595 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:08.210843 (+ 2248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:08.210851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:08.211332 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:08.212042 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:08.212063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.213956 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.213960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:08.215328 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:08.215341 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.216411 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.216415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:08.216595 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:08.217728 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:08.217746 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:08.219502 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:08.221986 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:08.221991 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:08.222003 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:08.222447 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:08.223482 (+ 1035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:08.223485 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:08.223488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:08.223588 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:08.224394 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:08.224397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:08.224506 (+ 109us) client_negotiation.cc:769] Sending connection context
0811 04:40:08.224643 (+ 137us) client_negotiation.cc:240] Negotiation successful
0811 04:40:08.224969 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":169,"spinlock_wait_cycles":2304}
I20260811 04:40:08.225236 21240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.208454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:47513 (local address 127.2.238.130:41465)
0811 04:40:08.208774 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:08.208808 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:08.208850 (+ 42us) server_negotiation.cc:407] Connection header received
0811 04:40:08.209920 (+ 1070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:08.209929 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:08.210415 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:08.210733 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:08.212228 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.213749 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.215451 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.216254 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.216460 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:08.219859 (+ 3399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:08.219865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:08.219878 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:08.219905 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:08.221855 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:08.222553 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:08.222566 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:08.222571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:08.223370 (+ 799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:08.223699 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:08.223704 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:08.223707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:08.224276 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:08.224405 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:08.224778 (+ 373us) server_negotiation.cc:299] Negotiation successful
0811 04:40:08.225058 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":118,"threads_started":1}
I20260811 04:40:08.225610 21242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.208443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:39867 (local address 127.2.238.131:37741)
0811 04:40:08.209330 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:08.209340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:08.209369 (+ 29us) server_negotiation.cc:407] Connection header received
0811 04:40:08.210227 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:08.210235 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:08.210663 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:08.210932 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:08.212228 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.213749 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.216471 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.217260 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.217495 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:08.220206 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:08.220210 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:08.220221 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:08.220271 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:08.222297 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:08.223009 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:08.223014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:08.223020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:08.223152 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:08.223599 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:08.223602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:08.223604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:08.224037 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:08.224189 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:08.225182 (+ 993us) server_negotiation.cc:299] Negotiation successful
0811 04:40:08.225428 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":406,"threads_started":1}
I20260811 04:40:08.225030 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.208718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:37741 (local address 127.2.238.129:39867)
0811 04:40:08.209039 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:08.209061 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:08.209158 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:08.211334 (+ 2176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:08.211336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:08.211484 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:08.212042 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:08.212063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.213956 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.213960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:08.216309 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:08.216321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:08.217413 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:08.217419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:08.217519 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:08.218009 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:08.218023 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:08.220011 (+ 1988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:08.222523 (+ 2512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:08.222528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:08.222533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:08.222822 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:08.223295 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:08.223299 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:08.223301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:08.223482 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:08.224179 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:08.224185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:08.224411 (+ 226us) client_negotiation.cc:769] Sending connection context
0811 04:40:08.224640 (+ 229us) client_negotiation.cc:240] Negotiation successful
0811 04:40:08.224890 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":81,"spinlock_wait_cycles":768,"thread_start_us":76,"threads_started":1}
W20260811 04:40:08.228354 20826 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
W20260811 04:40:08.228503 20827 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:40:08.228605 20827 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6464be8500143359c73bbb30fb7b704; no voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408
I20260811 04:40:08.228888 21239 raft_consensus.cc:2726] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:40:08.329775 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.329368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:55851)
0811 04:40:08.329595 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:08.329669 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260811 04:40:08.994402 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:08.993928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:46031)
0811 04:40:08.994230 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:08.994286 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":99,"threads_started":1}
I20260811 04:40:09.138283 21245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:09.137880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:42611)
0811 04:40:09.138112 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:09.138178 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260811 04:40:09.331127 21246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:09.330653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:36745)
0811 04:40:09.330943 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:09.331010 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":80,"threads_started":1}
I20260811 04:40:09.995620 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:09.995208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:51917)
0811 04:40:09.995471 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:09.995526 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
I20260811 04:40:10.139468 21248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:10.138988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:60757)
0811 04:40:10.139301 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:10.139371 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":72,"threads_started":1}
I20260811 04:40:10.152918 21249 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:10.153064 21249 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:10.153621 21249 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
W20260811 04:40:10.154475 20827 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
W20260811 04:40:10.154567 20826 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:40:10.154698 20826 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6464be8500143359c73bbb30fb7b704; no voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408
I20260811 04:40:10.154860 21249 raft_consensus.cc:2726] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:40:10.332347 21250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:10.331926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:44783)
0811 04:40:10.332182 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:10.332255 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260811 04:40:10.996793 21251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:10.996378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:54749)
0811 04:40:10.996636 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:10.996698 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":103,"threads_started":1}
I20260811 04:40:11.140651 21252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:11.140266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:60815)
0811 04:40:11.140509 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:11.140562 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260811 04:40:11.333536 21253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:11.333102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:43469)
0811 04:40:11.333383 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:11.333441 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":96,"threads_started":1}
I20260811 04:40:11.998376 21254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:11.997851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:44853)
0811 04:40:11.998207 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:11.998261 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":105,"threads_started":1}
I20260811 04:40:12.141901 21255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:12.141406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:58349)
0811 04:40:12.141674 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:12.141731 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260811 04:40:12.317411 21256 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:12.317625 21256 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:12.318132 21256 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
W20260811 04:40:12.318972 20827 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
W20260811 04:40:12.319089 20826 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:40:12.319213 20826 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6464be8500143359c73bbb30fb7b704; no voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408
I20260811 04:40:12.319388 21256 raft_consensus.cc:2726] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:40:12.334923 21257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:12.334345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:45099)
0811 04:40:12.334733 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:12.334810 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":113,"threads_started":1}
I20260811 04:40:12.999617 21258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:12.999183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:35553)
0811 04:40:12.999472 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:12.999523 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
I20260811 04:40:13.143038 21259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:13.142640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:60113)
0811 04:40:13.142896 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:13.142948 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260811 04:40:13.336261 21260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:13.335742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:56395)
0811 04:40:13.336053 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:13.336136 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":76,"threads_started":1}
I20260811 04:40:14.001102 21261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:14.000563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:39719)
0811 04:40:14.000927 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:14.000997 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":132,"threads_started":1}
I20260811 04:40:14.144207 21262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:14.143770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:54351)
0811 04:40:14.144054 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:14.144106 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":69,"threads_started":1}
I20260811 04:40:14.330269 21263 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:14.330400 21263 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:14.330884 21263 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
W20260811 04:40:14.331771 20826 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
W20260811 04:40:14.331911 20827 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:40:14.332003 20827 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6464be8500143359c73bbb30fb7b704; no voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408
I20260811 04:40:14.332163 21263 raft_consensus.cc:2726] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:40:14.337466 21264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:14.337018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:56081)
0811 04:40:14.337307 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:14.337361 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":86,"threads_started":1}
I20260811 04:40:15.002394 21266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:15.001982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:45461)
0811 04:40:15.002237 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:15.002301 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":78,"threads_started":1}
I20260811 04:40:15.145331 21267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:15.144920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:58163)
0811 04:40:15.145183 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:15.145239 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260811 04:40:15.338615 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:15.338241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:42277)
0811 04:40:15.338474 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:15.338527 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260811 04:40:16.004000 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:16.003409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:34141)
0811 04:40:16.003768 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:16.003852 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":109,"threads_started":1}
I20260811 04:40:16.146616 21271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:16.146122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:56727)
0811 04:40:16.146453 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:16.146512 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260811 04:40:16.339918 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:16.339465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:47871)
0811 04:40:16.339735 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:16.339801 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":89,"threads_started":1}
I20260811 04:40:17.005498 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:17.004951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:41315)
0811 04:40:17.005327 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:17.005396 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
I20260811 04:40:17.147912 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:17.147478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:34449)
0811 04:40:17.147768 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:17.147817 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":100,"threads_started":1}
I20260811 04:40:17.341173 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:17.340723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:53423)
0811 04:40:17.341005 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:17.341061 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":94,"threads_started":1}
I20260811 04:40:18.006872 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:18.006280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:41477)
0811 04:40:18.006648 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:18.006718 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260811 04:40:18.149147 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:18.148690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:37199)
0811 04:40:18.148979 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:18.149035 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":89,"threads_started":1}
I20260811 04:40:18.342356 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:18.341946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:48095)
0811 04:40:18.342201 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:18.342258 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":71,"threads_started":1}
I20260811 04:40:19.008286 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:19.007839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:47551)
0811 04:40:19.008119 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:19.008178 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":91,"threads_started":1}
I20260811 04:40:19.150358 21280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:19.149959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:54811)
0811 04:40:19.150213 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:19.150267 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":69,"threads_started":1}
I20260811 04:40:19.343535 21281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:19.343136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:47581)
0811 04:40:19.343367 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:19.343420 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260811 04:40:19.566375 21282 raft_consensus.cc:482] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:19.566514 21282 raft_consensus.cc:504] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21b8db948e9045aaabeb9467f7031d67" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 41465 } } peers { permanent_uuid: "ac3bc6f24a0746dcabe1a67c6f22f408" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 37741 } } peers { permanent_uuid: "c6464be8500143359c73bbb30fb7b704" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 44585 } }
I20260811 04:40:19.567020 21282 leader_election.cc:288] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465), ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741)
W20260811 04:40:19.567910 20826 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac3bc6f24a0746dcabe1a67c6f22f408 (127.2.238.131:37741): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
W20260811 04:40:19.568032 20827 leader_election.cc:341] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 21b8db948e9045aaabeb9467f7031d67 (127.2.238.130:41465): Not found: Tablet not found: 79d3cf58e1fc4d2db7f7b217d880afdf
I20260811 04:40:19.568136 20827 leader_election.cc:302] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6464be8500143359c73bbb30fb7b704; no voters: 21b8db948e9045aaabeb9467f7031d67, ac3bc6f24a0746dcabe1a67c6f22f408
I20260811 04:40:19.568305 21282 raft_consensus.cc:2726] T 79d3cf58e1fc4d2db7f7b217d880afdf P c6464be8500143359c73bbb30fb7b704 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260811 04:40:20.009528 21283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:20.009118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:58811)
0811 04:40:20.009364 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:20.009426 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":75,"threads_started":1}
I20260811 04:40:20.151613 21284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:20.151164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:40677)
0811 04:40:20.151449 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:20.151510 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":91,"threads_started":1}
I20260811 04:40:20.344796 21285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:20.344355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:38159)
0811 04:40:20.344638 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:20.344696 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260811 04:40:21.010720 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:21.010281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:53533)
0811 04:40:21.010546 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:21.010620 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":95,"threads_started":1}
I20260811 04:40:21.153005 21287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:21.152511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:37753)
0811 04:40:21.152844 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:21.152909 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":138,"threads_started":1}
I20260811 04:40:21.346166 21288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:21.345670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.131:56441)
0811 04:40:21.345987 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:21.346058 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":156,"threads_started":1}
I20260811 04:40:22.012032 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:22.011663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.129:50115)
0811 04:40:22.011891 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:22.011940 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260811 04:40:22.154433 21290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:22.154003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:39749 (local address 127.2.238.130:40971)
0811 04:40:22.154281 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:22.154335 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.190:39749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
I20260811 04:40:22.327582 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20764
I20260811 04:40:22.337105 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 20949
I20260811 04:40:22.345526 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21092
2026-08-11T04:40:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59095 ms)
[----------] 30 tests from SecurityITest (239834 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:22 dist-test-slave-0kls krb5kdc[21293](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:22 dist-test-slave-0kls krb5kdc[21293](info): set up 2 sockets
Aug 11 04:40:22 dist-test-slave-0kls krb5kdc[21293](info): commencing operation
krb5kdc: starting...
W20260811 04:40:23.394200 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:23 dist-test-slave-0kls krb5kdc[21293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:23.519032 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34299
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34299
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:23.633353 21306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:23.633598 21306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:23.633634 21306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:23.637564 21306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:23.637640 21306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:23.637660 21306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:23.637676 21306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:23.637691 21306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:23.642961 21306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:23.644062 21306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:23.645119 21306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:23.653993 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:23.654107 21306 server_base.cc:1034] running on GCE node
W20260811 04:40:23.654032 21316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:23.654028 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:23.654870 21306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:23.655876 21306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:23.657037 21306 hybrid_clock.cc:648] HybridClock initialized: now 1786423223657016 us; error 29 us; skew 500 ppm
Aug 11 04:40:23 dist-test-slave-0kls krb5kdc[21293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:23.663183 21306 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:23.664060 21306 webserver.cc:460] Webserver started at http://127.2.238.190:40023/ using document root <none> and password file <none>
I20260811 04:40:23.664567 21306 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:23.664624 21306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:23.664798 21306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:23.666850 21306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "34ed92a2eed143ac8f03b70508d33e28"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "cfb7b7d08caa23a079f74f125fa11328"
server_key_iv: "353a7920d4b88e8a08c4dad025dfe8a6"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.667351 21306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "34ed92a2eed143ac8f03b70508d33e28"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "cfb7b7d08caa23a079f74f125fa11328"
server_key_iv: "353a7920d4b88e8a08c4dad025dfe8a6"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.671010 21306 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260811 04:40:23.673309 21323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:23.674582 21306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:23.674736 21306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "34ed92a2eed143ac8f03b70508d33e28"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "cfb7b7d08caa23a079f74f125fa11328"
server_key_iv: "353a7920d4b88e8a08c4dad025dfe8a6"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.674852 21306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:23.688253 21306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:23.691300 21306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:23.691524 21306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:23.699064 21306 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34299
I20260811 04:40:23.699107 21378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34299 every 8 connection(s)
I20260811 04:40:23.700096 21306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:23.702996 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:23.706215 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21306
I20260811 04:40:23.706346 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:23.706676 3002 external_mini_cluster.cc:1428] Setting key e59d9dfaa680098a53dd6538758b3902
I20260811 04:40:23.709395 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: Bootstrap starting.
I20260811 04:40:23.711740 21380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:23.712500 21380 log.cc:824] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:23.714568 21380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: No bootstrap required, opened a new log
Aug 11 04:40:23 dist-test-slave-0kls krb5kdc[21293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:23.717779 21380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } }
I20260811 04:40:23.718086 21380 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:23.718181 21380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34ed92a2eed143ac8f03b70508d33e28, State: Initialized, Role: FOLLOWER
I20260811 04:40:23.718621 21380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } }
I20260811 04:40:23.718770 21380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:23.718888 21380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:23.719004 21380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:23.720077 21380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } }
I20260811 04:40:23.720463 21380 leader_election.cc:302] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34ed92a2eed143ac8f03b70508d33e28; no voters:
I20260811 04:40:23.720844 21380 leader_election.cc:288] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:23.720907 21387 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:23.721154 21387 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [term 1 LEADER]: Becoming Leader. State: Replica: 34ed92a2eed143ac8f03b70508d33e28, State: Running, Role: LEADER
I20260811 04:40:23.721478 21387 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } }
I20260811 04:40:23.721890 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:23.708201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.2.238.190:34299)
0811 04:40:23.708660 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:23.708676 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:23.708713 (+ 37us) server_negotiation.cc:407] Connection header received
0811 04:40:23.709324 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:23.709339 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:23.709751 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:23.710205 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:23.711072 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.712121 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.713056 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.713525 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.713706 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:23.716323 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:23.716328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:23.716341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:23.716368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:23.718725 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:23.719232 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:23.719237 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:23.719241 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:23.719346 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:23.719673 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:23.719675 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:23.719677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:23.720075 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:23.720257 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:23.720529 (+ 272us) server_negotiation.cc:299] Negotiation successful
0811 04:40:23.720889 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260811 04:40:23.722154 21380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:23.722652 21387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34ed92a2eed143ac8f03b70508d33e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } } }
I20260811 04:40:23.722855 21387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:23.724109 21394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:23.724248 21388 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34ed92a2eed143ac8f03b70508d33e28. Latest consensus state: current_term: 1 leader_uuid: "34ed92a2eed143ac8f03b70508d33e28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34ed92a2eed143ac8f03b70508d33e28" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34299 } } }
I20260811 04:40:23.724344 21388 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:23.727308 21394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:23.732831 21394 catalog_manager.cc:1347] Generated new cluster ID: adf2d11456c24470a419883f1d65a6ab
I20260811 04:40:23.732890 21394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:23.741745 21394 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:23.743052 21394 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:23.750094 21394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: Generated new TSK 0
I20260811 04:40:23.750777 21394 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:40:23.793190 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:23.911101 21409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:23.911406 21409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:23.911480 21409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:23.915606 21409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:23.915699 21409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:23.915793 21409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:23.921423 21409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:23.922724 21409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:23.923934 21409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:23.933609 21415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:23.933683 21414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:23.933759 21418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:23.934092 21409 server_base.cc:1034] running on GCE node
I20260811 04:40:23.934501 21409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:23.935231 21409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:23.936422 21409 hybrid_clock.cc:648] HybridClock initialized: now 1786423223936388 us; error 45 us; skew 500 ppm
Aug 11 04:40:23 dist-test-slave-0kls krb5kdc[21293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:23.939229 21409 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:40:23.940157 21409 webserver.cc:460] Webserver started at http://127.2.238.129:46013/ using document root <none> and password file <none>
I20260811 04:40:23.940701 21409 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:23.940759 21409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:23.940924 21409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:23.942997 21409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "90534415eb0141baabe9eb9d104bb9f2"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "93ba3305051142020ba6c1566c38e75f"
server_key_iv: "1538d305670745447c09c05825b6710f"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.943593 21409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "90534415eb0141baabe9eb9d104bb9f2"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "93ba3305051142020ba6c1566c38e75f"
server_key_iv: "1538d305670745447c09c05825b6710f"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.947472 21409 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:40:23.950076 21425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:23.951285 21409 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:23.951401 21409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "90534415eb0141baabe9eb9d104bb9f2"
format_stamp: "Formatted at 2026-08-11 04:40:23 on dist-test-slave-0kls"
server_key: "93ba3305051142020ba6c1566c38e75f"
server_key_iv: "1538d305670745447c09c05825b6710f"
server_key_version: "encryptionkey@0"
I20260811 04:40:23.951522 21409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:23.963351 21409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:23.966123 21409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:23.966320 21409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:23.966984 21409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:23.967796 21409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:23.967844 21409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:23.967905 21409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:23.967938 21409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:23.977270 21409 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:40807
I20260811 04:40:23.977324 21546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:40807 every 8 connection(s)
I20260811 04:40:23.978313 21409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:23.980499 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21409
I20260811 04:40:23.980623 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:40:23.981003 3002 external_mini_cluster.cc:1428] Setting key b990192f2f3b6828218ceb7c4612cd75
Aug 11 04:40:23 dist-test-slave-0kls krb5kdc[21293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423223, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:40:23.991993 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:23.980196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:33505 (local address 127.2.238.190:34299)
0811 04:40:23.980349 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:23.980360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:23.981451 (+ 1091us) server_negotiation.cc:407] Connection header received
0811 04:40:23.982461 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:23.982464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:23.982514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:23.982611 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:23.983767 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.984299 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.985406 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.985620 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.985726 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:23.988497 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:23.988501 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:23.988502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:23.988524 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:23.989929 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:23.990508 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:23.990510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:23.990511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:23.990579 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:23.990926 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:23.990928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:23.990929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:23.991097 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:23.991168 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:23.991736 (+ 568us) server_negotiation.cc:299] Negotiation successful
0811 04:40:23.991880 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 04:40:23.992887 21549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:23.980716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34299 (local address 127.2.238.129:33505)
0811 04:40:23.981302 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:23.981344 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:23.982273 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:23.982779 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:23.982785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:23.983145 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:23.983617 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:23.983632 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.984436 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.984440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:23.985281 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:23.985290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:23.985765 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:23.985769 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:23.985955 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:23.986506 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:23.986521 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:23.988320 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:23.990050 (+ 1730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:23.990056 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:23.990070 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:23.990417 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:23.990664 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:23.990668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:23.990672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:23.990833 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:23.991205 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:23.991213 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:23.991513 (+ 300us) client_negotiation.cc:769] Sending connection context
0811 04:40:23.991707 (+ 194us) client_negotiation.cc:240] Negotiation successful
0811 04:40:23.992040 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":164,"threads_started":1}
I20260811 04:40:23.994202 21547 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34299
I20260811 04:40:23.994441 21547 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
I20260811 04:40:23.995045 21547 heartbeater.cc:511] Master 127.2.238.190:34299 requested a full tablet report, sending...
I20260811 04:40:23.996554 21341 ts_manager.cc:194] Registered new tserver with Master: 90534415eb0141baabe9eb9d104bb9f2 (127.2.238.129:40807)
I20260811 04:40:23.997835 21341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:33505
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:40:24.017901 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:24.129763 21552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:24.130025 21552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:24.130066 21552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:24.134011 21552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:24.134102 21552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:24.134208 21552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:24.139137 21552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:24.140105 21552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:24.141063 21552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:24.150574 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:24.150739 21552 server_base.cc:1034] running on GCE node
W20260811 04:40:24.150588 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:24.150627 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:24.151300 21552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:24.151916 21552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:24.153124 21552 hybrid_clock.cc:648] HybridClock initialized: now 1786423224153103 us; error 40 us; skew 500 ppm
Aug 11 04:40:24 dist-test-slave-0kls krb5kdc[21293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:24.156164 21552 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:40:24.157035 21552 webserver.cc:460] Webserver started at http://127.2.238.130:41379/ using document root <none> and password file <none>
I20260811 04:40:24.157609 21552 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:24.157702 21552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:24.157963 21552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:24.159957 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "f59f8b7eeef74c0eb354aaac07a3924c"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "196dd982dd9671c026241a260a45b4e0"
server_key_iv: "074437302a8da0fa8e91644c04a8faf1"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.160531 21552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "f59f8b7eeef74c0eb354aaac07a3924c"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "196dd982dd9671c026241a260a45b4e0"
server_key_iv: "074437302a8da0fa8e91644c04a8faf1"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.164285 21552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:40:24.166728 21569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.167904 21552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:24.168058 21552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "f59f8b7eeef74c0eb354aaac07a3924c"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "196dd982dd9671c026241a260a45b4e0"
server_key_iv: "074437302a8da0fa8e91644c04a8faf1"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.168174 21552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:24.184890 21552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:24.188153 21552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:24.188338 21552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:24.188895 21552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:24.189738 21552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:24.189829 21552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.189904 21552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:24.189988 21552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.198274 21552 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:36261
I20260811 04:40:24.198299 21694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:36261 every 8 connection(s)
I20260811 04:40:24.199157 21552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:24.205385 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21552
I20260811 04:40:24.205525 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:24.205852 3002 external_mini_cluster.cc:1428] Setting key 3347f3a8f7bc5bea0c0e300c206f9eca
Aug 11 04:40:24 dist-test-slave-0kls krb5kdc[21293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:24.212836 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.200979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:58713 (local address 127.2.238.190:34299)
0811 04:40:24.201119 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.201123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.202017 (+ 894us) server_negotiation.cc:407] Connection header received
0811 04:40:24.202893 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.202896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.202951 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.203048 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:24.204148 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.204679 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.206180 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.206421 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.206533 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.208879 (+ 2346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:24.208882 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:24.208884 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:24.208907 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:24.210593 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.211196 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.211199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.211200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.211276 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.211637 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.211639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.211640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.211826 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:24.211900 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.212559 (+ 659us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.212711 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:40:24.213955 21697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.201334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34299 (local address 127.2.238.130:58713)
0811 04:40:24.201872 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.201920 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.202697 (+ 777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.203218 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.203225 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.203536 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:24.203991 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.204005 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.204809 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.204812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.206056 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.206067 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.206583 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.206587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.206767 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.207401 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:24.207417 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:24.208730 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:24.210717 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:24.210725 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:24.210752 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:24.211086 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:24.211363 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:24.211368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:24.211373 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:24.211546 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:24.211926 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:24.211935 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:24.212252 (+ 317us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.212510 (+ 258us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.212887 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":127,"threads_started":1}
I20260811 04:40:24.215185 21695 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34299
I20260811 04:40:24.215423 21695 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:24.215891 21695 heartbeater.cc:511] Master 127.2.238.190:34299 requested a full tablet report, sending...
I20260811 04:40:24.216888 21341 ts_manager.cc:194] Registered new tserver with Master: f59f8b7eeef74c0eb354aaac07a3924c (127.2.238.130:36261)
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:40:24.217453 21341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:58713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:40:24.236304 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:24.347455 21700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:24.347710 21700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:24.347793 21700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:24.351601 21700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:24.351679 21700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:24.351775 21700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:24.356909 21700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:24.358044 21700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:24.359131 21700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:24.368921 21706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:24.369055 21700 server_base.cc:1034] running on GCE node
W20260811 04:40:24.368911 21711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:24.368911 21707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:24.369764 21700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:24.370467 21700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:24.371639 21700 hybrid_clock.cc:648] HybridClock initialized: now 1786423224371623 us; error 47 us; skew 500 ppm
Aug 11 04:40:24 dist-test-slave-0kls krb5kdc[21293](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:24.374583 21700 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:40:24.375488 21700 webserver.cc:460] Webserver started at http://127.2.238.131:34733/ using document root <none> and password file <none>
I20260811 04:40:24.376065 21700 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:24.376128 21700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:24.376305 21700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:24.378494 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "d4d675971528d6cf6f158614524f37b0"
server_key_iv: "1b5c0eb01de459df403fd569a4549d5a"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.379029 21700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "d4d675971528d6cf6f158614524f37b0"
server_key_iv: "1b5c0eb01de459df403fd569a4549d5a"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.383045 21700 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:40:24.385432 21719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.386750 21700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:40:24.386884 21700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "d4d675971528d6cf6f158614524f37b0"
server_key_iv: "1b5c0eb01de459df403fd569a4549d5a"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.386987 21700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:24.404150 21700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:24.406966 21700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:24.407173 21700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:24.407757 21700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:24.408550 21700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:24.408615 21700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.408692 21700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:24.408726 21700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.417340 21700 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:34111
I20260811 04:40:24.417362 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:34111 every 8 connection(s)
I20260811 04:40:24.418342 21700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:24.423306 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21700
I20260811 04:40:24.423439 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:24.423777 3002 external_mini_cluster.cc:1428] Setting key fefc5fbd3f02fce5453fac3e78651d9a
Aug 11 04:40:24 dist-test-slave-0kls krb5kdc[21293](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423224, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:24.431818 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.420170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:51929 (local address 127.2.238.190:34299)
0811 04:40:24.420320 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.420323 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.421187 (+ 864us) server_negotiation.cc:407] Connection header received
0811 04:40:24.422108 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.422111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.422167 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.422265 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:24.423570 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.424191 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.425688 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.426027 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.426141 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.428658 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:24.428661 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:24.428663 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:24.428690 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:24.430020 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.430503 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.430505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.430506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.430584 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.430874 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.430876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.430877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.431051 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:24.431126 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.431589 (+ 463us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.431698 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:40:24.432652 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.420557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34299 (local address 127.2.238.131:51929)
0811 04:40:24.421038 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.421080 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.421898 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.422434 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.422440 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.422770 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:24.423395 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.423420 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.424362 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.424366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.425562 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.425572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.426187 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.426191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.426367 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.426910 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:24.426926 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:24.428465 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:24.430125 (+ 1660us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:24.430129 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:24.430142 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:24.430413 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:24.430664 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:24.430666 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:24.430668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:24.430797 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:24.431142 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:24.431147 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:24.431378 (+ 231us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.431544 (+ 166us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.431801 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260811 04:40:24.433858 21841 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34299
I20260811 04:40:24.434118 21841 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:24.434621 21841 heartbeater.cc:511] Master 127.2.238.190:34299 requested a full tablet report, sending...
I20260811 04:40:24.435726 21341 ts_manager.cc:194] Registered new tserver with Master: 44eaf47c86b1403abd8fcfe0dbe7c7d2 (127.2.238.131:34111)
I20260811 04:40:24.436399 21341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:51929
I20260811 04:40:24.438006 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:24.447463 21385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.440118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54760 (local address 127.2.238.190:34299)
0811 04:40:24.440381 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.440384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.440397 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:40:24.440514 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.440516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.440554 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.440611 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:24.441334 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.441922 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.442717 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.442912 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.442983 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.444462 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:24.444465 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:24.444467 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:24.444487 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:24.445755 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.446297 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.446300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.446301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.446378 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:24.446614 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:24.446616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:24.446617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:24.446823 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:24.446925 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.447163 (+ 238us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.447258 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260811 04:40:24.450639 21340 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:24.453773 21340 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:24.467711 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.462525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.2.238.131:34111)
0811 04:40:24.463528 (+ 1003us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.463544 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.463568 (+ 24us) server_negotiation.cc:407] Connection header received
0811 04:40:24.463640 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.463644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.463820 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.463975 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:24.464360 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.465276 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.466630 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.467162 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.467283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467416 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.467487 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.467577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":129,"threads_started":1}
I20260811 04:40:24.467690 21854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.461980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:34111 (local address 127.0.0.1:57698)
0811 04:40:24.463079 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.463092 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.463204 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.464124 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.464127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.464157 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:24.464245 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.464250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.465429 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.465433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.466520 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.466527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.467285 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.467288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.467365 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467386 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.467463 (+ 77us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.467532 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":26,"thread_start_us":91,"threads_started":1}
I20260811 04:40:24.467856 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.462461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:36261 (local address 127.0.0.1:46542)
0811 04:40:24.463315 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.463327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.463444 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.463819 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.463823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.463839 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:24.463962 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.463969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.465033 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.465037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.466290 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.466301 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.467494 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.467497 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.467587 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467623 (+ 36us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.467690 (+ 67us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.467748 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"mutex_wait_us":22,"thread_start_us":77,"threads_started":1}
I20260811 04:40:24.467865 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.461911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52274 (local address 127.2.238.129:40807)
0811 04:40:24.462342 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.462359 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.462384 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:40:24.462462 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.462467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.462684 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.462822 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:24.463354 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.464467 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.466082 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.466775 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.466876 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467033 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.467614 (+ 581us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.467720 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":134,"threads_started":1}
I20260811 04:40:24.468318 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.462896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.2.238.130:36261)
0811 04:40:24.463337 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.463349 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.463432 (+ 83us) server_negotiation.cc:407] Connection header received
0811 04:40:24.463546 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.463551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.463697 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.463843 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:24.464058 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.464886 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.466417 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.467194 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.467286 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467417 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.468081 (+ 664us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.468157 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":120,"threads_started":1}
I20260811 04:40:24.468083 21852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.461749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40807 (local address 127.0.0.1:52274)
0811 04:40:24.462136 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.462198 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.462327 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.463035 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.463042 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.463066 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:24.463212 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.463220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.464620 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.464623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.465957 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.465967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.467095 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.467098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.467206 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.467260 (+ 54us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.467354 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.467491 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":104,"threads_started":1}
I20260811 04:40:24.471261 21769 tablet_service.cc:1467] Processing CreateTablet for tablet 12c78f0b096d48988835a9f465ffbecb (DEFAULT_TABLE table=test-table [id=3d6e202486d04ec3ad876fd1217d2b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:24.471261 21479 tablet_service.cc:1467] Processing CreateTablet for tablet 12c78f0b096d48988835a9f465ffbecb (DEFAULT_TABLE table=test-table [id=3d6e202486d04ec3ad876fd1217d2b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:24.471253 21623 tablet_service.cc:1467] Processing CreateTablet for tablet 12c78f0b096d48988835a9f465ffbecb (DEFAULT_TABLE table=test-table [id=3d6e202486d04ec3ad876fd1217d2b81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:24.472352 21623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c78f0b096d48988835a9f465ffbecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:24.472378 21479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c78f0b096d48988835a9f465ffbecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:24.472383 21769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c78f0b096d48988835a9f465ffbecb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:24.478864 21858 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Bootstrap starting.
I20260811 04:40:24.480377 21860 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Bootstrap starting.
I20260811 04:40:24.480551 21859 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: Bootstrap starting.
I20260811 04:40:24.481601 21858 tablet_bootstrap.cc:654] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:24.482422 21858 log.cc:824] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:24.482616 21859 tablet_bootstrap.cc:654] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:24.483088 21860 tablet_bootstrap.cc:654] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:24.483903 21860 log.cc:824] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:24.483903 21859 log.cc:824] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:24.484786 21858 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: No bootstrap required, opened a new log
I20260811 04:40:24.484993 21858 ts_tablet_manager.cc:1397] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:40:24.485852 21859 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: No bootstrap required, opened a new log
I20260811 04:40:24.486032 21859 ts_tablet_manager.cc:1397] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:40:24.486083 21860 tablet_bootstrap.cc:492] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: No bootstrap required, opened a new log
I20260811 04:40:24.486304 21860 ts_tablet_manager.cc:1397] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:40:24.488024 21858 raft_consensus.cc:348] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.488298 21858 raft_consensus.cc:374] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:24.488381 21858 raft_consensus.cc:729] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f59f8b7eeef74c0eb354aaac07a3924c, State: Initialized, Role: FOLLOWER
I20260811 04:40:24.488830 21858 consensus_queue.cc:260] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.489464 21695 heartbeater.cc:503] Master 127.2.238.190:34299 was elected leader, sending a full tablet report...
I20260811 04:40:24.489765 21858 ts_tablet_manager.cc:1428] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:40:24.489887 21860 raft_consensus.cc:348] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.490231 21860 raft_consensus.cc:374] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:24.490307 21860 raft_consensus.cc:729] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44eaf47c86b1403abd8fcfe0dbe7c7d2, State: Initialized, Role: FOLLOWER
I20260811 04:40:24.490705 21860 consensus_queue.cc:260] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.490936 21859 raft_consensus.cc:348] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.491219 21859 raft_consensus.cc:374] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:24.491304 21859 raft_consensus.cc:729] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90534415eb0141baabe9eb9d104bb9f2, State: Initialized, Role: FOLLOWER
I20260811 04:40:24.491360 21841 heartbeater.cc:503] Master 127.2.238.190:34299 was elected leader, sending a full tablet report...
I20260811 04:40:24.491694 21860 ts_tablet_manager.cc:1428] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260811 04:40:24.491796 21859 consensus_queue.cc:260] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.492568 21547 heartbeater.cc:503] Master 127.2.238.190:34299 was elected leader, sending a full tablet report...
I20260811 04:40:24.492856 21859 ts_tablet_manager.cc:1428] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:40:24.621177 21865 raft_consensus.cc:482] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:24.621428 21865 raft_consensus.cc:504] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.622927 21865 leader_election.cc:288] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90534415eb0141baabe9eb9d104bb9f2 (127.2.238.129:40807), f59f8b7eeef74c0eb354aaac07a3924c (127.2.238.130:36261)
I20260811 04:40:24.627676 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.623236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40807 (local address 127.2.238.131:57761)
0811 04:40:24.623406 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.623425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.623537 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.623916 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.623919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.623939 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:24.624087 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.624093 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.625084 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.625087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.626342 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.626354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.627203 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.627206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.627314 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.627340 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.627418 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.627496 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260811 04:40:24.627682 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.623568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:36261 (local address 127.2.238.131:48283)
0811 04:40:24.623963 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.623975 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:24.624072 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:24.624349 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:24.624351 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:24.624366 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:24.624461 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.624465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.625343 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.625346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.626513 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:24.626521 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.627336 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.627339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:24.627438 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.627456 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:40:24.627518 (+ 62us) client_negotiation.cc:240] Negotiation successful
0811 04:40:24.627568 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":2,"thread_start_us":148,"threads_started":1}
I20260811 04:40:24.627823 21853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.623317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:57761 (local address 127.2.238.129:40807)
0811 04:40:24.623464 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.623470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.623497 (+ 27us) server_negotiation.cc:407] Connection header received
0811 04:40:24.623716 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.623721 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.623785 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.623872 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:24.624199 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.624969 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.626464 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.627084 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.627209 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.627268 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.627451 (+ 183us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.627549 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260811 04:40:24.627866 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.623691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:48283 (local address 127.2.238.130:36261)
0811 04:40:24.623828 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.623833 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.624064 (+ 231us) server_negotiation.cc:407] Connection header received
0811 04:40:24.624190 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.624193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.624244 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.624326 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:24.624572 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.625240 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.626615 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.627228 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.627337 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.627371 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.627638 (+ 267us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.627739 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:40:24.628449 21499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c78f0b096d48988835a9f465ffbecb" candidate_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534415eb0141baabe9eb9d104bb9f2" is_pre_election: true
I20260811 04:40:24.628454 21643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c78f0b096d48988835a9f465ffbecb" candidate_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" is_pre_election: true
I20260811 04:40:24.628752 21499 raft_consensus.cc:2454] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44eaf47c86b1403abd8fcfe0dbe7c7d2 in term 0.
I20260811 04:40:24.628773 21643 raft_consensus.cc:2454] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44eaf47c86b1403abd8fcfe0dbe7c7d2 in term 0.
I20260811 04:40:24.629251 21720 leader_election.cc:302] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44eaf47c86b1403abd8fcfe0dbe7c7d2, 90534415eb0141baabe9eb9d104bb9f2; no voters:
I20260811 04:40:24.629498 21865 raft_consensus.cc:2781] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:24.629586 21865 raft_consensus.cc:482] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:24.629631 21865 raft_consensus.cc:3037] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:24.630744 21865 raft_consensus.cc:504] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.631094 21865 leader_election.cc:288] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [CANDIDATE]: Term 1 election: Requested vote from peers 90534415eb0141baabe9eb9d104bb9f2 (127.2.238.129:40807), f59f8b7eeef74c0eb354aaac07a3924c (127.2.238.130:36261)
I20260811 04:40:24.631477 21499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c78f0b096d48988835a9f465ffbecb" candidate_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90534415eb0141baabe9eb9d104bb9f2"
I20260811 04:40:24.631501 21643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c78f0b096d48988835a9f465ffbecb" candidate_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f59f8b7eeef74c0eb354aaac07a3924c"
I20260811 04:40:24.631623 21643 raft_consensus.cc:3037] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:24.631613 21499 raft_consensus.cc:3037] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:24.632951 21643 raft_consensus.cc:2454] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44eaf47c86b1403abd8fcfe0dbe7c7d2 in term 1.
I20260811 04:40:24.632951 21499 raft_consensus.cc:2454] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44eaf47c86b1403abd8fcfe0dbe7c7d2 in term 1.
I20260811 04:40:24.633285 21720 leader_election.cc:302] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44eaf47c86b1403abd8fcfe0dbe7c7d2, f59f8b7eeef74c0eb354aaac07a3924c; no voters:
I20260811 04:40:24.633456 21865 raft_consensus.cc:2781] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:24.633700 21865 raft_consensus.cc:686] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 1 LEADER]: Becoming Leader. State: Replica: 44eaf47c86b1403abd8fcfe0dbe7c7d2, State: Running, Role: LEADER
I20260811 04:40:24.634104 21865 consensus_queue.cc:237] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } }
I20260811 04:40:24.637468 21341 catalog_manager.cc:5495] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 44eaf47c86b1403abd8fcfe0dbe7c7d2 (127.2.238.131). New cstate: current_term: 1 leader_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "44eaf47c86b1403abd8fcfe0dbe7c7d2" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 34111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 } health_report { overall_health: UNKNOWN } } }
I20260811 04:40:24.655781 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.652073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.2.238.131:34111)
0811 04:40:24.652228 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.652233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.652248 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:40:24.652411 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.652414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.652469 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.652580 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:24.652831 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.653345 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.654290 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.654507 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.654638 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.654930 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:24.655343 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:24.655479 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.655543 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.655599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:40:24.661653 21499 raft_consensus.cc:1261] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 1 FOLLOWER]: Refusing update from remote peer 44eaf47c86b1403abd8fcfe0dbe7c7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:24.661661 21643 raft_consensus.cc:1261] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 1 FOLLOWER]: Refusing update from remote peer 44eaf47c86b1403abd8fcfe0dbe7c7d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:24.662384 21865 consensus_queue.cc:1035] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f59f8b7eeef74c0eb354aaac07a3924c" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:24.662621 21868 consensus_queue.cc:1035] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90534415eb0141baabe9eb9d104bb9f2" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 40807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:24.677733 21340 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:24.677951 21340 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:24.680348 21340 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 34ed92a2eed143ac8f03b70508d33e28: Sending DeleteTablet for 3 replicas of tablet 12c78f0b096d48988835a9f465ffbecb
I20260811 04:40:24.681223 21769 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c78f0b096d48988835a9f465ffbecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:24 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:57698
I20260811 04:40:24.681200 21479 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c78f0b096d48988835a9f465ffbecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:24 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52274
I20260811 04:40:24.681734 21881 tablet_replica.cc:330] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2: stopping tablet replica
I20260811 04:40:24.681886 21623 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c78f0b096d48988835a9f465ffbecb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:24 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:46542
I20260811 04:40:24.682091 21881 raft_consensus.cc:2229] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:24.682426 21881 raft_consensus.cc:2258] T 12c78f0b096d48988835a9f465ffbecb P 90534415eb0141baabe9eb9d104bb9f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:24.682548 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21409
I20260811 04:40:24.689957 21883 tablet_replica.cc:330] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: stopping tablet replica
I20260811 04:40:24.690346 21883 raft_consensus.cc:2229] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:24.690667 21883 raft_consensus.cc:2258] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:24.694037 21882 tablet_replica.cc:330] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: stopping tablet replica
I20260811 04:40:24.694370 21882 raft_consensus.cc:2229] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:24.694823 21882 raft_consensus.cc:2258] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:24.695621 21883 ts_tablet_manager.cc:1905] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:24.696209 21882 ts_tablet_manager.cc:1905] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260811 04:40:24.698323 21324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:40807 (error 108)
I20260811 04:40:24.698984 21882 ts_tablet_manager.cc:1918] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:24.699076 21882 log.cc:1196] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/12c78f0b096d48988835a9f465ffbecb
I20260811 04:40:24.699397 21882 ts_tablet_manager.cc:1939] T 12c78f0b096d48988835a9f465ffbecb P 44eaf47c86b1403abd8fcfe0dbe7c7d2: Deleting consensus metadata
I20260811 04:40:24.699929 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.699539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:40807 (local address 127.0.0.1:52288)
0811 04:40:24.699737 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.699845 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260811 04:40:24.700107 21324 catalog_manager.cc:4584] TS 90534415eb0141baabe9eb9d104bb9f2 (127.2.238.129:40807): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12c78f0b096d48988835a9f465ffbecb: Network error: Client connection negotiation failed: client connection to 127.2.238.129:40807: connect: Connection refused (error 111)
I20260811 04:40:24.700428 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21552
I20260811 04:40:24.700578 21883 ts_tablet_manager.cc:1918] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:24.700695 21883 log.cc:1196] T 12c78f0b096d48988835a9f465ffbecb P f59f8b7eeef74c0eb354aaac07a3924c: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/12c78f0b096d48988835a9f465ffbecb
I20260811 04:40:24.700676 21325 catalog_manager.cc:4841] TS 44eaf47c86b1403abd8fcfe0dbe7c7d2 (127.2.238.131:34111): tablet 12c78f0b096d48988835a9f465ffbecb (table test-table [id=3d6e202486d04ec3ad876fd1217d2b81]) successfully deleted
I20260811 04:40:24.713729 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21700
I20260811 04:40:24.714185 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.713771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:36261 (local address 127.0.0.1:46544)
0811 04:40:24.714021 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:24.714085 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.130:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260811 04:40:24.723887 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21306
2026-08-11T04:40:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2377 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-11T04:40:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:24Z Disabled control of system clock
I20260811 04:40:24.757869 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:33583
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:33583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:24.870854 21889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:24.871114 21889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:24.871155 21889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:24.874905 21889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:24.874955 21889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:24.874971 21889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:24.874989 21889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:24.875003 21889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:24.879549 21889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:33583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:33583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:24.880606 21889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:24.882018 21889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:40:24.889992 21889 server_base.cc:1034] running on GCE node
W20260811 04:40:24.889961 21897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:24.889925 21895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:24.889945 21894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:24.890836 21889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:24.891849 21889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:24.893006 21889 hybrid_clock.cc:648] HybridClock initialized: now 1786423224892982 us; error 33 us; skew 500 ppm
I20260811 04:40:24.894758 21889 webserver.cc:460] Webserver started at http://127.2.238.190:44431/ using document root <none> and password file <none>
I20260811 04:40:24.895328 21889 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:24.895385 21889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:24.895649 21889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:24.897478 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "cbc47ce54de94abc8344e02910460799"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "46e14f8c7be8801588e0fb2a40dcb744"
server_key_iv: "023c1d455bba61eb3e03b5669209f923"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.898038 21889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "cbc47ce54de94abc8344e02910460799"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "46e14f8c7be8801588e0fb2a40dcb744"
server_key_iv: "023c1d455bba61eb3e03b5669209f923"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.901530 21889 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:40:24.903946 21904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:24.905143 21889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:24.905267 21889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "cbc47ce54de94abc8344e02910460799"
format_stamp: "Formatted at 2026-08-11 04:40:24 on dist-test-slave-0kls"
server_key: "46e14f8c7be8801588e0fb2a40dcb744"
server_key_iv: "023c1d455bba61eb3e03b5669209f923"
server_key_version: "encryptionkey@0"
I20260811 04:40:24.905367 21889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:24.916924 21889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:24.917567 21889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:24.917773 21889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:24.924913 21889 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:33583
I20260811 04:40:24.924925 21961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:33583 every 8 connection(s)
I20260811 04:40:24.925963 21889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:24.929152 21963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:24.933458 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21889
I20260811 04:40:24.933584 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:24.933892 3002 external_mini_cluster.cc:1428] Setting key 6ccb65a651c2aa3fa2cad1006af69d6e
I20260811 04:40:24.935165 21963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: Bootstrap starting.
I20260811 04:40:24.937490 21963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:24.938298 21963 log.cc:824] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:24.940099 21963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: No bootstrap required, opened a new log
I20260811 04:40:24.943145 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:24.935439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43210 (local address 127.2.238.190:33583)
0811 04:40:24.936154 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:24.936171 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:24.936207 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:40:24.936807 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:24.936822 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:24.937212 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:24.937532 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:24.937990 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.938982 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.939863 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:24.940412 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:24.940594 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:24.941452 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:24.941457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:24.941469 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:24.941892 (+ 423us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:24.941944 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:24.941963 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:24.942057 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:24.942230 (+ 173us) server_negotiation.cc:299] Negotiation successful
0811 04:40:24.942475 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":163,"threads_started":1}
I20260811 04:40:24.943686 21963 raft_consensus.cc:348] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } }
I20260811 04:40:24.943961 21963 raft_consensus.cc:374] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:24.944025 21963 raft_consensus.cc:729] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cbc47ce54de94abc8344e02910460799, State: Initialized, Role: FOLLOWER
I20260811 04:40:24.944435 21963 consensus_queue.cc:260] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } }
I20260811 04:40:24.944553 21963 raft_consensus.cc:388] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:24.944597 21963 raft_consensus.cc:482] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:24.944674 21963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:24.945541 21963 raft_consensus.cc:504] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } }
I20260811 04:40:24.945883 21963 leader_election.cc:302] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cbc47ce54de94abc8344e02910460799; no voters:
I20260811 04:40:24.946128 21963 leader_election.cc:288] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:24.946244 21968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:24.946506 21968 raft_consensus.cc:686] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [term 1 LEADER]: Becoming Leader. State: Replica: cbc47ce54de94abc8344e02910460799, State: Running, Role: LEADER
I20260811 04:40:24.946805 21968 consensus_queue.cc:237] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } }
I20260811 04:40:24.947432 21963 sys_catalog.cc:564] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:24.948475 21969 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cbc47ce54de94abc8344e02910460799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } } }
I20260811 04:40:24.948477 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cbc47ce54de94abc8344e02910460799. Latest consensus state: current_term: 1 leader_uuid: "cbc47ce54de94abc8344e02910460799" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cbc47ce54de94abc8344e02910460799" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 33583 } } }
I20260811 04:40:24.948637 21969 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:24.948639 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:24.949743 21978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:24.952696 21978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:24.957420 21978 catalog_manager.cc:1347] Generated new cluster ID: 44f6384d476547439ad6e8d13d26f31f
I20260811 04:40:24.957492 21978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:24.964550 21978 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:24.965402 21978 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:24.974032 21978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: Generated new TSK 0
I20260811 04:40:24.974728 21978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:24.984798 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--builtin_ntp_servers=127.2.238.148:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:25.097301 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:25.097520 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:25.097553 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:25.101389 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:25.101437 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:25.101519 21989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:25.106192 21989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:25.107247 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:25.108634 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:25.117977 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.117971 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.117983 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:25.118338 21989 server_base.cc:1034] running on GCE node
I20260811 04:40:25.118743 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:25.119370 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:25.120529 21989 hybrid_clock.cc:648] HybridClock initialized: now 1786423225120499 us; error 41 us; skew 500 ppm
I20260811 04:40:25.122366 21989 webserver.cc:460] Webserver started at http://127.2.238.129:33981/ using document root <none> and password file <none>
I20260811 04:40:25.122921 21989 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:25.122982 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:25.123230 21989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:25.125274 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "c301e802867d444198300039e07c5567"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "f95facfd693cd4d83e9c4d948c011676"
server_key_iv: "a0f7d2807dd06b4aa5321f9eb4e33866"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.125903 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "c301e802867d444198300039e07c5567"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "f95facfd693cd4d83e9c4d948c011676"
server_key_iv: "a0f7d2807dd06b4aa5321f9eb4e33866"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.129709 21989 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260811 04:40:25.132230 22004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.133492 21989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:25.133642 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "c301e802867d444198300039e07c5567"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "f95facfd693cd4d83e9c4d948c011676"
server_key_iv: "a0f7d2807dd06b4aa5321f9eb4e33866"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.133764 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:25.156028 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:25.156782 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:25.157011 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:25.157697 21989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:25.158730 21989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:25.158802 21989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.158877 21989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:25.158921 21989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.168013 21989 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:42519
I20260811 04:40:25.168045 22125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:42519 every 8 connection(s)
I20260811 04:40:25.168947 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:25.171903 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 21989
I20260811 04:40:25.172021 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:40:25.172353 3002 external_mini_cluster.cc:1428] Setting key d37586d74316fef214b667bea62b3c5c
I20260811 04:40:25.174520 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--builtin_ntp_servers=127.2.238.148:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:25.179379 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.170914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:55685 (local address 127.2.238.190:33583)
0811 04:40:25.171047 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.171051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.172273 (+ 1222us) server_negotiation.cc:407] Connection header received
0811 04:40:25.173958 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.173960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.174021 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.174086 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:25.175312 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.175866 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.177173 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.177574 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.177717 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.178830 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:25.178837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:25.178842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:25.178862 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:25.178890 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:25.178905 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:25.179001 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.179129 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.179201 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:40:25.180227 22128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.171314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:33583 (local address 127.2.238.129:55685)
0811 04:40:25.172089 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.172157 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.173711 (+ 1554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.174282 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.174292 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.174404 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:25.175164 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.175188 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.176004 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.176011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.177047 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.177058 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.177757 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.177759 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.177949 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.178570 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:25.178572 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:25.178611 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:25.178675 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:25.178994 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:25.179002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:25.179006 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.179097 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.179386 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":106,"threads_started":1}
I20260811 04:40:25.181537 22126 heartbeater.cc:347] Connected to a master server at 127.2.238.190:33583
I20260811 04:40:25.181777 22126 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:25.182339 22126 heartbeater.cc:511] Master 127.2.238.190:33583 requested a full tablet report, sending...
I20260811 04:40:25.184028 21923 ts_manager.cc:194] Registered new tserver with Master: c301e802867d444198300039e07c5567 (127.2.238.129:42519)
I20260811 04:40:25.185750 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.129:55685
W20260811 04:40:25.286895 22129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:25.287163 22129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:25.287204 22129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:25.291370 22129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:25.291426 22129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:25.291528 22129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:25.296258 22129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22129
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:25.297286 22129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:25.298722 22129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:25.308512 22135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.308563 22134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.308609 22137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:25.308655 22129 server_base.cc:1034] running on GCE node
I20260811 04:40:25.309154 22129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:25.309907 22129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:25.311100 22129 hybrid_clock.cc:648] HybridClock initialized: now 1786423225311072 us; error 54 us; skew 500 ppm
I20260811 04:40:25.312907 22129 webserver.cc:460] Webserver started at http://127.2.238.130:45077/ using document root <none> and password file <none>
I20260811 04:40:25.313485 22129 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:25.313580 22129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:25.313846 22129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:25.316092 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "494da54a105a4324bcb2e2ad1a7e4fe9"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "812b6eff20010cf719ba9bb891ec5285"
server_key_iv: "29f3b188047636b3a5e086fc037cc72d"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.316715 22129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "494da54a105a4324bcb2e2ad1a7e4fe9"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "812b6eff20010cf719ba9bb891ec5285"
server_key_iv: "29f3b188047636b3a5e086fc037cc72d"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.320534 22129 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260811 04:40:25.323156 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.324385 22129 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:40:25.324524 22129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "494da54a105a4324bcb2e2ad1a7e4fe9"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "812b6eff20010cf719ba9bb891ec5285"
server_key_iv: "29f3b188047636b3a5e086fc037cc72d"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.324644 22129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:25.341095 22129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:25.341768 22129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:25.342005 22129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:25.342576 22129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:25.343489 22129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:25.343559 22129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.343631 22129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:25.343675 22129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.353096 22129 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:42693
I20260811 04:40:25.353132 22266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:42693 every 8 connection(s)
I20260811 04:40:25.354089 22129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:25.362151 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22129
I20260811 04:40:25.362304 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:25.362627 3002 external_mini_cluster.cc:1428] Setting key ab0144d50a2b26dd3390b192bbc678af
I20260811 04:40:25.363489 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.355973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:42753 (local address 127.2.238.190:33583)
0811 04:40:25.356154 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.356158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.357069 (+ 911us) server_negotiation.cc:407] Connection header received
0811 04:40:25.358536 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.358539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.358607 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.358745 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:25.359674 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.360257 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.361456 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.361677 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.361813 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.362989 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:25.362993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:25.362994 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:25.363007 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:25.363029 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:25.363043 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:25.363120 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.363281 (+ 161us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.363344 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260811 04:40:25.364415 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.356277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:33583 (local address 127.2.238.130:42753)
0811 04:40:25.356886 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.356925 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.358307 (+ 1382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.358926 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.358932 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.359027 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:25.359502 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.359515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.360416 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.360423 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.361320 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.361329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.361868 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.361873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.362067 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.362723 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:25.362725 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:25.362763 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:25.362822 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:25.363122 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:25.363134 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:25.363143 (+ 9us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.363275 (+ 132us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.363521 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260811 04:40:25.364588 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--builtin_ntp_servers=127.2.238.148:37969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:25.365635 22267 heartbeater.cc:347] Connected to a master server at 127.2.238.190:33583
I20260811 04:40:25.365943 22267 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:25.366465 22267 heartbeater.cc:511] Master 127.2.238.190:33583 requested a full tablet report, sending...
I20260811 04:40:25.367570 21923 ts_manager.cc:194] Registered new tserver with Master: 494da54a105a4324bcb2e2ad1a7e4fe9 (127.2.238.130:42693)
I20260811 04:40:25.368186 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.130:42753
W20260811 04:40:25.475958 22270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:25.476226 22270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:25.476296 22270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:25.480262 22270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:25.480341 22270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:25.480427 22270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:25.485109 22270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:37969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:33583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:25.486295 22270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:25.487809 22270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:25.497015 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.497102 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:25.497040 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:25.497213 22270 server_base.cc:1034] running on GCE node
I20260811 04:40:25.497684 22270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:25.498318 22270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:25.499488 22270 hybrid_clock.cc:648] HybridClock initialized: now 1786423225499469 us; error 27 us; skew 500 ppm
I20260811 04:40:25.501297 22270 webserver.cc:460] Webserver started at http://127.2.238.131:43249/ using document root <none> and password file <none>
I20260811 04:40:25.501967 22270 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:25.502085 22270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:25.502367 22270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:25.504490 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "ffd28214b9f446c3b8287863736cd81d"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "30c4211692002ea604947c6776602c1a"
server_key_iv: "526259a8a358cc1f64342b8f9a9ee224"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.505074 22270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "ffd28214b9f446c3b8287863736cd81d"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "30c4211692002ea604947c6776602c1a"
server_key_iv: "526259a8a358cc1f64342b8f9a9ee224"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.508690 22270 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:40:25.511162 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.512290 22270 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:40:25.512432 22270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "ffd28214b9f446c3b8287863736cd81d"
format_stamp: "Formatted at 2026-08-11 04:40:25 on dist-test-slave-0kls"
server_key: "30c4211692002ea604947c6776602c1a"
server_key_iv: "526259a8a358cc1f64342b8f9a9ee224"
server_key_version: "encryptionkey@0"
I20260811 04:40:25.512533 22270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:25.538116 22270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:25.538830 22270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:25.539011 22270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:25.539660 22270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:25.540618 22270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:25.540665 22270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.540731 22270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:25.540764 22270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:25.550109 22270 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:42087
I20260811 04:40:25.550150 22409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:42087 every 8 connection(s)
I20260811 04:40:25.550995 22270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:25.551781 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22270
I20260811 04:40:25.551909 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:25.552184 3002 external_mini_cluster.cc:1428] Setting key 1aee0b3cb82a048c2ebe564d5c4a0630
I20260811 04:40:25.559849 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.552829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:47885 (local address 127.2.238.190:33583)
0811 04:40:25.553105 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.553110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.554127 (+ 1017us) server_negotiation.cc:407] Connection header received
0811 04:40:25.555473 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.555476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.555522 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.555596 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:25.556439 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.556961 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.558099 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.558313 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.558389 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.559444 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:25.559447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:25.559449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:25.559459 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:25.559474 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:25.559480 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:25.559531 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.559667 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.559712 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260811 04:40:25.560667 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.553230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:33583 (local address 127.2.238.131:47885)
0811 04:40:25.553974 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.554026 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.555284 (+ 1258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.555764 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.555784 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.555863 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:25.556324 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.556336 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.557062 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.557065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.557999 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.558008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.558442 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.558445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.558575 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.559213 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:25.559215 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:25.559252 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:25.559309 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:25.559550 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:25.559557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:25.559561 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.559659 (+ 98us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.559917 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":136,"threads_started":1}
I20260811 04:40:25.561815 22410 heartbeater.cc:347] Connected to a master server at 127.2.238.190:33583
I20260811 04:40:25.562031 22410 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:25.562496 22410 heartbeater.cc:511] Master 127.2.238.190:33583 requested a full tablet report, sending...
I20260811 04:40:25.563467 21923 ts_manager.cc:194] Registered new tserver with Master: ffd28214b9f446c3b8287863736cd81d (127.2.238.131:42087)
I20260811 04:40:25.563987 21923 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.131:47885
I20260811 04:40:25.566313 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:25.572588 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.568378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43218 (local address 127.2.238.190:33583)
0811 04:40:25.568546 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.568550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.569255 (+ 705us) server_negotiation.cc:407] Connection header received
0811 04:40:25.569333 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.569335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.569373 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.569436 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:25.569720 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.570325 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.571103 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.571305 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.571384 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.572161 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:25.572164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:25.572166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:25.572177 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:25.572194 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:25.572201 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:25.572258 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.572354 (+ 96us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.572401 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260811 04:40:25.575389 21923 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43218:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:25.578331 21923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:25.592741 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.587370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55852 (local address 127.2.238.129:42519)
0811 04:40:25.587777 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.587788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.587811 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:40:25.587902 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.587908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.588096 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.588471 (+ 375us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:25.588623 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589750 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.591361 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.592090 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.592215 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.592415 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.592478 (+ 63us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.592591 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
I20260811 04:40:25.592729 22424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.586905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:42519 (local address 127.0.0.1:55852)
0811 04:40:25.587387 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.587400 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.587493 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.588268 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.588274 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.588306 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:25.588483 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.588490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589943 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.589949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.591256 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.591268 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.592216 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.592218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.592301 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.592330 (+ 29us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.592440 (+ 110us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.592571 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":167,"spinlock_wait_cycles":2432,"thread_start_us":83,"threads_started":1}
I20260811 04:40:25.592729 22427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.587308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:42087 (local address 127.0.0.1:47348)
0811 04:40:25.588079 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.588091 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.588236 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.588842 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.588844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.588859 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:25.588914 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.588917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589836 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.589840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.591443 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.591454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.592409 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.592415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.592496 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.592515 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.592580 (+ 65us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.592626 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"mutex_wait_us":707,"thread_start_us":81,"threads_started":1}
I20260811 04:40:25.592849 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.587550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.2.238.131:42087)
0811 04:40:25.587930 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.587941 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.588163 (+ 222us) server_negotiation.cc:407] Connection header received
0811 04:40:25.588375 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.588380 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.588557 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.588691 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:25.589020 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589706 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.591571 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.592289 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.592387 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.592544 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.592596 (+ 52us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.592690 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":155,"threads_started":1}
I20260811 04:40:25.594950 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.586558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:42693 (local address 127.0.0.1:58862)
0811 04:40:25.587029 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.587080 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.587218 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.588269 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.588277 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.588313 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:25.588620 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.588629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589961 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.589964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.591443 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.591454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.594520 (+ 3066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.594524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.594619 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.594654 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.594748 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.594813 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":132,"spinlock_wait_cycles":5504,"thread_start_us":108,"threads_started":1}
I20260811 04:40:25.595507 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.586759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58862 (local address 127.2.238.130:42693)
0811 04:40:25.587549 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.587561 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.587586 (+ 25us) server_negotiation.cc:407] Connection header received
0811 04:40:25.587666 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.587673 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.587917 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.588073 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:25.588745 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.589831 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.593622 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.594380 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.594498 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.594682 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.595228 (+ 546us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.595337 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":140,"threads_started":1}
I20260811 04:40:25.595927 22339 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 (DEFAULT_TABLE table=test-table [id=1ab189228c3f4c348beb012e828a8e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:25.595948 22054 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 (DEFAULT_TABLE table=test-table [id=1ab189228c3f4c348beb012e828a8e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:25.596980 22054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6ad093dd9e4443bf0b09110085ed86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:25.597003 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6ad093dd9e4443bf0b09110085ed86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:25.598161 22195 tablet_service.cc:1467] Processing CreateTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 (DEFAULT_TABLE table=test-table [id=1ab189228c3f4c348beb012e828a8e3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:25.599031 22195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f6ad093dd9e4443bf0b09110085ed86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:25.603091 22428 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Bootstrap starting.
I20260811 04:40:25.605618 22430 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Bootstrap starting.
I20260811 04:40:25.606284 22428 tablet_bootstrap.cc:654] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:25.607146 22428 log.cc:824] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:25.607755 22430 tablet_bootstrap.cc:654] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:25.607854 22429 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Bootstrap starting.
I20260811 04:40:25.608611 22430 log.cc:824] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:25.609328 22428 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: No bootstrap required, opened a new log
I20260811 04:40:25.609517 22428 ts_tablet_manager.cc:1397] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260811 04:40:25.610121 22429 tablet_bootstrap.cc:654] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:25.610746 22429 log.cc:824] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:25.612685 22429 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: No bootstrap required, opened a new log
I20260811 04:40:25.612895 22429 ts_tablet_manager.cc:1397] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 04:40:25.612929 22430 tablet_bootstrap.cc:492] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: No bootstrap required, opened a new log
I20260811 04:40:25.613131 22430 ts_tablet_manager.cc:1397] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260811 04:40:25.613494 22428 raft_consensus.cc:348] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.613777 22428 raft_consensus.cc:374] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:25.613860 22428 raft_consensus.cc:729] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c301e802867d444198300039e07c5567, State: Initialized, Role: FOLLOWER
I20260811 04:40:25.614321 22428 consensus_queue.cc:260] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.615003 22126 heartbeater.cc:503] Master 127.2.238.190:33583 was elected leader, sending a full tablet report...
I20260811 04:40:25.615321 22428 ts_tablet_manager.cc:1428] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 04:40:25.616887 22430 raft_consensus.cc:348] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.617197 22430 raft_consensus.cc:374] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:25.617268 22430 raft_consensus.cc:729] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494da54a105a4324bcb2e2ad1a7e4fe9, State: Initialized, Role: FOLLOWER
I20260811 04:40:25.617107 22429 raft_consensus.cc:348] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.617352 22429 raft_consensus.cc:374] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:25.617424 22429 raft_consensus.cc:729] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd28214b9f446c3b8287863736cd81d, State: Initialized, Role: FOLLOWER
I20260811 04:40:25.617740 22430 consensus_queue.cc:260] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.617877 22429 consensus_queue.cc:260] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.618667 22429 ts_tablet_manager.cc:1428] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:40:25.618716 22430 ts_tablet_manager.cc:1428] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:40:25.619007 22410 heartbeater.cc:503] Master 127.2.238.190:33583 was elected leader, sending a full tablet report...
I20260811 04:40:25.619007 22267 heartbeater.cc:503] Master 127.2.238.190:33583 was elected leader, sending a full tablet report...
W20260811 04:40:25.670233 22127 tablet.cc:2369] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:40:25.685225 22435 raft_consensus.cc:482] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:25.685469 22435 raft_consensus.cc:504] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.686628 22435 leader_election.cc:288] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffd28214b9f446c3b8287863736cd81d (127.2.238.131:42087), c301e802867d444198300039e07c5567 (127.2.238.129:42519)
I20260811 04:40:25.690459 22269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.686885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:42519 (local address 127.2.238.130:41987)
0811 04:40:25.687057 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.687073 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.687168 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.687434 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.687437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.687456 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:25.687585 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.687593 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.688471 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.688474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.689324 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.689334 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.690041 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.690046 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.690148 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.690180 (+ 32us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.690251 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.690311 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":1}
I20260811 04:40:25.690486 22425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.687030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:41987 (local address 127.2.238.129:42519)
0811 04:40:25.687186 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.687192 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.687212 (+ 20us) server_negotiation.cc:407] Connection header received
0811 04:40:25.687267 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.687271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.687335 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.687422 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:25.687743 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.688369 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.689443 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.689930 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.690018 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.690059 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.690283 (+ 224us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.690365 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260811 04:40:25.690518 22437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.686885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:42087 (local address 127.2.238.130:55813)
0811 04:40:25.687429 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.687441 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:25.687580 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:25.687878 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:25.687881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:25.687898 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:25.688009 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.688015 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.688713 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.688716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.689486 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:25.689496 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.690156 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.690158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:25.690244 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.690270 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:40:25.690341 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 04:40:25.690394 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"mutex_wait_us":138,"thread_start_us":156,"threads_started":1}
I20260811 04:40:25.690546 22426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.687031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:55813 (local address 127.2.238.131:42087)
0811 04:40:25.687214 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.687221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.687555 (+ 334us) server_negotiation.cc:407] Connection header received
0811 04:40:25.687717 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.687720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.687772 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.687829 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:25.688109 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.688612 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.689600 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.690041 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.690167 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.690209 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.690369 (+ 160us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.690433 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260811 04:40:25.691084 22076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6ad093dd9e4443bf0b09110085ed86" candidate_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c301e802867d444198300039e07c5567" is_pre_election: true
I20260811 04:40:25.691082 22360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6ad093dd9e4443bf0b09110085ed86" candidate_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd28214b9f446c3b8287863736cd81d" is_pre_election: true
I20260811 04:40:25.691388 22360 raft_consensus.cc:2454] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 494da54a105a4324bcb2e2ad1a7e4fe9 in term 0.
I20260811 04:40:25.691397 22076 raft_consensus.cc:2454] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 494da54a105a4324bcb2e2ad1a7e4fe9 in term 0.
I20260811 04:40:25.691897 22149 leader_election.cc:302] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 494da54a105a4324bcb2e2ad1a7e4fe9, c301e802867d444198300039e07c5567, ffd28214b9f446c3b8287863736cd81d; no voters:
I20260811 04:40:25.692202 22435 raft_consensus.cc:2781] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:25.692273 22435 raft_consensus.cc:482] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:25.692301 22435 raft_consensus.cc:3037] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:25.693426 22435 raft_consensus.cc:504] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.693791 22435 leader_election.cc:288] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [CANDIDATE]: Term 1 election: Requested vote from peers ffd28214b9f446c3b8287863736cd81d (127.2.238.131:42087), c301e802867d444198300039e07c5567 (127.2.238.129:42519)
I20260811 04:40:25.694186 22360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6ad093dd9e4443bf0b09110085ed86" candidate_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffd28214b9f446c3b8287863736cd81d"
I20260811 04:40:25.694177 22076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f6ad093dd9e4443bf0b09110085ed86" candidate_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c301e802867d444198300039e07c5567"
I20260811 04:40:25.694303 22360 raft_consensus.cc:3037] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:25.694304 22076 raft_consensus.cc:3037] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:25.695580 22360 raft_consensus.cc:2454] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494da54a105a4324bcb2e2ad1a7e4fe9 in term 1.
I20260811 04:40:25.695580 22076 raft_consensus.cc:2454] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494da54a105a4324bcb2e2ad1a7e4fe9 in term 1.
I20260811 04:40:25.695868 22144 leader_election.cc:302] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494da54a105a4324bcb2e2ad1a7e4fe9, c301e802867d444198300039e07c5567; no voters:
I20260811 04:40:25.696019 22435 raft_consensus.cc:2781] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:25.696236 22435 raft_consensus.cc:686] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 1 LEADER]: Becoming Leader. State: Replica: 494da54a105a4324bcb2e2ad1a7e4fe9, State: Running, Role: LEADER
I20260811 04:40:25.696547 22435 consensus_queue.cc:237] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } }
I20260811 04:40:25.699922 21923 catalog_manager.cc:5495] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 494da54a105a4324bcb2e2ad1a7e4fe9 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494da54a105a4324bcb2e2ad1a7e4fe9" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 42693 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 } health_report { overall_health: UNKNOWN } } }
I20260811 04:40:25.715597 22423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.712126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.2.238.130:42693)
0811 04:40:25.712351 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:25.712355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:25.712368 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:40:25.712436 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:25.712438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:25.712492 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:25.712594 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:25.712804 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.713310 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.714220 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:25.714431 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:25.714503 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:25.714761 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:25.715220 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:25.715312 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:25.715396 (+ 84us) server_negotiation.cc:299] Negotiation successful
0811 04:40:25.715460 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260811 04:40:25.721629 22360 raft_consensus.cc:1261] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 1 FOLLOWER]: Refusing update from remote peer 494da54a105a4324bcb2e2ad1a7e4fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:25.721629 22076 raft_consensus.cc:1261] T 7f6ad093dd9e4443bf0b09110085ed86 P c301e802867d444198300039e07c5567 [term 1 FOLLOWER]: Refusing update from remote peer 494da54a105a4324bcb2e2ad1a7e4fe9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:25.722471 22435 consensus_queue.cc:1035] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffd28214b9f446c3b8287863736cd81d" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 42087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:25.722702 22439 consensus_queue.cc:1035] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c301e802867d444198300039e07c5567" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:25.730931 22443 mvcc.cc:205] Tried to move back new op lower bound from 7317189532547747840 to 7317189532455526400. Current Snapshot: MvccSnapshot[applied={T|T < 7317189532547747840}]
I20260811 04:40:25.740159 21924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:25.740298 21924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43218:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:25.742715 21924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cbc47ce54de94abc8344e02910460799: Sending DeleteTablet for 3 replicas of tablet 7f6ad093dd9e4443bf0b09110085ed86
I20260811 04:40:25.743513 22054 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:25 UTC) from {username='slave'} at 127.0.0.1:55852
I20260811 04:40:25.743516 22339 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:25 UTC) from {username='slave'} at 127.0.0.1:47348
I20260811 04:40:25.743527 22195 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f6ad093dd9e4443bf0b09110085ed86 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:25 UTC) from {username='slave'} at 127.0.0.1:58862
I20260811 04:40:25.744040 22452 tablet_replica.cc:330] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: stopping tablet replica
I20260811 04:40:25.744347 22452 raft_consensus.cc:2229] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:25.744602 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21989
I20260811 04:40:25.744666 22452 raft_consensus.cc:2258] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:25.745015 22455 tablet_replica.cc:330] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: stopping tablet replica
I20260811 04:40:25.745332 22455 raft_consensus.cc:2229] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:25.745771 22455 raft_consensus.cc:2258] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:25.747990 22452 ts_tablet_manager.cc:1905] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:25.751183 22452 ts_tablet_manager.cc:1918] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:25.751312 22452 log.cc:1196] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/7f6ad093dd9e4443bf0b09110085ed86
I20260811 04:40:25.751600 22452 ts_tablet_manager.cc:1939] T 7f6ad093dd9e4443bf0b09110085ed86 P ffd28214b9f446c3b8287863736cd81d: Deleting consensus metadata
I20260811 04:40:25.753963 21908 catalog_manager.cc:4841] TS ffd28214b9f446c3b8287863736cd81d (127.2.238.131:42087): tablet 7f6ad093dd9e4443bf0b09110085ed86 (table test-table [id=1ab189228c3f4c348beb012e828a8e3b]) successfully deleted
I20260811 04:40:25.755160 22455 ts_tablet_manager.cc:1905] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:25.757218 22455 ts_tablet_manager.cc:1918] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:25.757342 22455 log.cc:1196] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/7f6ad093dd9e4443bf0b09110085ed86
I20260811 04:40:25.757632 22455 ts_tablet_manager.cc:1939] T 7f6ad093dd9e4443bf0b09110085ed86 P 494da54a105a4324bcb2e2ad1a7e4fe9: Deleting consensus metadata
W20260811 04:40:25.758397 21905 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:42519 (error 108)
I20260811 04:40:25.758852 21908 catalog_manager.cc:4841] TS 494da54a105a4324bcb2e2ad1a7e4fe9 (127.2.238.130:42693): tablet 7f6ad093dd9e4443bf0b09110085ed86 (table test-table [id=1ab189228c3f4c348beb012e828a8e3b]) successfully deleted
I20260811 04:40:25.759240 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22129
I20260811 04:40:25.760710 22422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:25.759434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:42519 (local address 127.0.0.1:55856)
0811 04:40:25.760450 (+ 1016us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:25.760526 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":923}
W20260811 04:40:25.761720 21905 catalog_manager.cc:4584] TS c301e802867d444198300039e07c5567 (127.2.238.129:42519): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f6ad093dd9e4443bf0b09110085ed86: Network error: Client connection negotiation failed: client connection to 127.2.238.129:42519: connect: Connection refused (error 111)
I20260811 04:40:25.767303 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22270
I20260811 04:40:25.775388 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 21889
2026-08-11T04:40:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1050 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:25 dist-test-slave-0kls krb5kdc[22459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:25 dist-test-slave-0kls krb5kdc[22459](info): set up 2 sockets
Aug 11 04:40:25 dist-test-slave-0kls krb5kdc[22459](info): commencing operation
krb5kdc: starting...
W20260811 04:40:26.829938 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:26 dist-test-slave-0kls krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:26.954592 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34483
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:27.062217 22473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:27.062484 22473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:27.062523 22473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:27.066545 22473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:27.066596 22473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:27.066622 22473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:27.066660 22473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:27.066692 22473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:27.071825 22473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:27.072789 22473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:27.073778 22473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:27.081948 22479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:27.082152 22473 server_base.cc:1034] running on GCE node
W20260811 04:40:27.081981 22480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.081955 22482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:27.082787 22473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:27.083752 22473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:27.084913 22473 hybrid_clock.cc:648] HybridClock initialized: now 1786423227084885 us; error 43 us; skew 500 ppm
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:27.087430 22473 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:27.088177 22473 webserver.cc:460] Webserver started at http://127.2.238.190:46805/ using document root <none> and password file <none>
I20260811 04:40:27.088636 22473 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:27.088687 22473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:27.088847 22473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:27.090723 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "06dd2dd715a847e09bcb16f187b18e87"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "425c1c46d06ccfa6da4d4ecf5802a7fe"
server_key_iv: "b4ae3e75bdd3a96da026ad1ed6ae6c96"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.091219 22473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "06dd2dd715a847e09bcb16f187b18e87"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "425c1c46d06ccfa6da4d4ecf5802a7fe"
server_key_iv: "b4ae3e75bdd3a96da026ad1ed6ae6c96"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.094769 22473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:40:27.096949 22489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.098186 22473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:27.098327 22473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "06dd2dd715a847e09bcb16f187b18e87"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "425c1c46d06ccfa6da4d4ecf5802a7fe"
server_key_iv: "b4ae3e75bdd3a96da026ad1ed6ae6c96"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.098425 22473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:27.112994 22473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:27.115857 22473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:27.116055 22473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:27.123179 22473 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34483
I20260811 04:40:27.123200 22546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34483 every 8 connection(s)
I20260811 04:40:27.124102 22473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:27.126979 22547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:27.130327 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22473
I20260811 04:40:27.130514 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:27.130842 3002 external_mini_cluster.cc:1428] Setting key 6876366cfa46e58cf06764e572288dd4
I20260811 04:40:27.132622 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: Bootstrap starting.
I20260811 04:40:27.135357 22547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:27.136083 22547 log.cc:824] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:27.138254 22547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: No bootstrap required, opened a new log
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:27.143030 22547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } }
I20260811 04:40:27.143337 22547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:27.143402 22547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06dd2dd715a847e09bcb16f187b18e87, State: Initialized, Role: FOLLOWER
I20260811 04:40:27.143901 22547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } }
I20260811 04:40:27.144068 22547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:27.144161 22547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:27.144263 22547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:27.145398 22547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } }
I20260811 04:40:27.145808 22547 leader_election.cc:302] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06dd2dd715a847e09bcb16f187b18e87; no voters:
I20260811 04:40:27.146121 22547 leader_election.cc:288] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:27.146236 22552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:27.146427 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.132235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57378 (local address 127.2.238.190:34483)
0811 04:40:27.132676 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.132692 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.132749 (+ 57us) server_negotiation.cc:407] Connection header received
0811 04:40:27.133583 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.133602 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.133994 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.134288 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:27.135250 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.136353 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.137394 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.137929 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.138057 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.140917 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:27.140923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:27.140939 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:27.140974 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:27.143019 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.143580 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.143585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.143590 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.143740 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.144038 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.144041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.144043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.144600 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:27.144788 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.145070 (+ 282us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.145408 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":169,"threads_started":1}
I20260811 04:40:27.147423 22547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:27.146476 22552 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [term 1 LEADER]: Becoming Leader. State: Replica: 06dd2dd715a847e09bcb16f187b18e87, State: Running, Role: LEADER
I20260811 04:40:27.147969 22552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } }
I20260811 04:40:27.149663 22553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06dd2dd715a847e09bcb16f187b18e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } } }
I20260811 04:40:27.149729 22554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06dd2dd715a847e09bcb16f187b18e87. Latest consensus state: current_term: 1 leader_uuid: "06dd2dd715a847e09bcb16f187b18e87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06dd2dd715a847e09bcb16f187b18e87" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34483 } } }
I20260811 04:40:27.149998 22553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:27.150003 22554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87 [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:27.152232 22567 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:27.152328 22567 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:27.152403 22568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:27.153165 22568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:27.158915 22568 catalog_manager.cc:1347] Generated new cluster ID: 7896dd887ebe4b4681ef257c888fe1e2
I20260811 04:40:27.158989 22568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:27.178997 22568 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:27.180253 22568 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:27.185423 22568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: Generated new TSK 0
I20260811 04:40:27.186226 22568 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:40:27.218842 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:27.338790 22574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:27.339044 22574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:27.339084 22574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:27.342988 22574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:27.343037 22574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:27.343091 22574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:27.348171 22574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:27.349200 22574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:27.350216 22574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:27.359472 22582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.359440 22579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.359488 22580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:27.359562 22574 server_base.cc:1034] running on GCE node
I20260811 04:40:27.360036 22574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:27.360611 22574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:27.361773 22574 hybrid_clock.cc:648] HybridClock initialized: now 1786423227361755 us; error 29 us; skew 500 ppm
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:27.364127 22574 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:40:27.364912 22574 webserver.cc:460] Webserver started at http://127.2.238.129:35553/ using document root <none> and password file <none>
I20260811 04:40:27.365456 22574 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:27.365537 22574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:27.365764 22574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:27.367659 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "b4c71eeeb1824dfea6073a133f4b5a60"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "1b74b3afdaa324929bdc1bf6371e966e"
server_key_iv: "308f729ed62e3e63f4aa4f1ff9aa2e22"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.368186 22574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "b4c71eeeb1824dfea6073a133f4b5a60"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "1b74b3afdaa324929bdc1bf6371e966e"
server_key_iv: "308f729ed62e3e63f4aa4f1ff9aa2e22"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.371801 22574 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:40:27.374156 22590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.375278 22574 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:27.375411 22574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "b4c71eeeb1824dfea6073a133f4b5a60"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "1b74b3afdaa324929bdc1bf6371e966e"
server_key_iv: "308f729ed62e3e63f4aa4f1ff9aa2e22"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.375526 22574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:27.386819 22574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:27.389410 22574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:27.389595 22574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:27.390162 22574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:27.390923 22574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:27.390987 22574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.391052 22574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:27.391099 22574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.399529 22574 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:33817
I20260811 04:40:27.399552 22715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:33817 every 8 connection(s)
I20260811 04:40:27.400375 22574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:27.405661 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22574
I20260811 04:40:27.405781 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:40:27.406056 3002 external_mini_cluster.cc:1428] Setting key 315e9985f0890eb8b1f631dc1d34bc44
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:27.414052 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.402271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:56333 (local address 127.2.238.190:34483)
0811 04:40:27.402456 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.402461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.403243 (+ 782us) server_negotiation.cc:407] Connection header received
0811 04:40:27.403975 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.403978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.404024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.404124 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:27.405219 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.405828 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.407097 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.407423 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.407541 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.410215 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:27.410218 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:27.410220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:27.410243 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:27.412023 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.412510 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.412516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.412521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.412629 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.412972 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.412976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.412979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.413215 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:27.413318 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.413761 (+ 443us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.413933 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260811 04:40:27.415208 22718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.402694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34483 (local address 127.2.238.129:56333)
0811 04:40:27.403099 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.403143 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.403796 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.404298 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.404304 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.404627 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:27.405072 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.405085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.405941 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.405944 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.406989 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.407000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.407584 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.407587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.407723 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.408320 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:27.408336 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:27.410055 (+ 1719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:27.412140 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.412144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.412154 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.412405 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.412735 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.412738 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.412741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.412879 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.413352 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:27.413362 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:27.413563 (+ 201us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.413714 (+ 151us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.414064 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":98,"threads_started":1}
I20260811 04:40:27.416378 22716 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34483
I20260811 04:40:27.416637 22716 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:27.417222 22716 heartbeater.cc:511] Master 127.2.238.190:34483 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
I20260811 04:40:27.418828 22507 ts_manager.cc:194] Registered new tserver with Master: b4c71eeeb1824dfea6073a133f4b5a60 (127.2.238.129:33817)
I20260811 04:40:27.420012 22507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:56333
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:40:27.436924 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:27.544564 22721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:27.544848 22721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:27.544924 22721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:27.548753 22721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:27.548823 22721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:27.548949 22721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:27.554020 22721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:27.555019 22721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:27.556016 22721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:27.565083 22727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.565102 22728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.565223 22730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:27.565320 22721 server_base.cc:1034] running on GCE node
I20260811 04:40:27.565901 22721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:27.566457 22721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:27.567667 22721 hybrid_clock.cc:648] HybridClock initialized: now 1786423227567658 us; error 35 us; skew 500 ppm
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:27.570101 22721 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:40:27.570909 22721 webserver.cc:460] Webserver started at http://127.2.238.130:41169/ using document root <none> and password file <none>
I20260811 04:40:27.571424 22721 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:27.571511 22721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:27.571730 22721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:27.573750 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "5deebf666c28436a8cee28303cded602"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "0abd217e84e1ae6f1f1a759ca84fe803"
server_key_iv: "65fcf16c2b416e95f32787620f738d11"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.574388 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "5deebf666c28436a8cee28303cded602"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "0abd217e84e1ae6f1f1a759ca84fe803"
server_key_iv: "65fcf16c2b416e95f32787620f738d11"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.578007 22721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:40:27.580484 22737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.581669 22721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:27.581833 22721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "5deebf666c28436a8cee28303cded602"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "0abd217e84e1ae6f1f1a759ca84fe803"
server_key_iv: "65fcf16c2b416e95f32787620f738d11"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.581948 22721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:27.601608 22721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:27.604476 22721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:27.604681 22721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:27.605265 22721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:27.606113 22721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:27.606181 22721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.606262 22721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:27.606302 22721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.614763 22721 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:33751
I20260811 04:40:27.614799 22859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:33751 every 8 connection(s)
I20260811 04:40:27.615617 22721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:27.623703 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22721
I20260811 04:40:27.623862 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:27.624199 3002 external_mini_cluster.cc:1428] Setting key 20970b54aecb844535305fb68265c229
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:27.628552 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.617260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:49215 (local address 127.2.238.190:34483)
0811 04:40:27.617389 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.617393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.618331 (+ 938us) server_negotiation.cc:407] Connection header received
0811 04:40:27.619326 (+ 995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.619329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.619372 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.619444 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:27.620500 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.621027 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.622110 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.622329 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.622425 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.624901 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:27.624905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:27.624908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:27.624938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:27.626581 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.627036 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.627042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.627047 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.627144 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.627434 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.627439 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.627443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.627672 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:27.627770 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.628351 (+ 581us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.628442 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:40:27.629551 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.617632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34483 (local address 127.2.238.130:49215)
0811 04:40:27.618166 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.618225 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.619145 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.619560 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.619566 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.619911 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:27.620369 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.620384 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.621148 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.621151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.621997 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.622006 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.622471 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.622473 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.622600 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.623071 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:27.623086 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:27.624743 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:27.626696 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.626700 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.626711 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.626943 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.627230 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.627232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.627234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.627349 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.627784 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:27.627793 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:27.628090 (+ 297us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.628292 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.628631 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":189,"threads_started":1}
I20260811 04:40:27.630764 22860 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34483
I20260811 04:40:27.630980 22860 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:27.631417 22860 heartbeater.cc:511] Master 127.2.238.190:34483 requested a full tablet report, sending...
I20260811 04:40:27.632488 22507 ts_manager.cc:194] Registered new tserver with Master: 5deebf666c28436a8cee28303cded602 (127.2.238.130:33751)
I20260811 04:40:27.633314 22507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:49215
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:40:27.653028 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:27.764200 22865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:27.764451 22865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:27.764541 22865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:27.768524 22865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:27.768574 22865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:27.768692 22865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:27.773736 22865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:27.774808 22865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:27.775821 22865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:27.785307 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.785312 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:27.785285 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:27.785506 22865 server_base.cc:1034] running on GCE node
I20260811 04:40:27.786067 22865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:27.786675 22865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:27.787874 22865 hybrid_clock.cc:648] HybridClock initialized: now 1786423227787847 us; error 42 us; skew 500 ppm
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:27.790683 22865 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:40:27.791488 22865 webserver.cc:460] Webserver started at http://127.2.238.131:45957/ using document root <none> and password file <none>
I20260811 04:40:27.792006 22865 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:27.792091 22865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:27.792327 22865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:27.794442 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "8df78697cde44e61b4a566117fa5af5b"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "cac42d1452010696444e4133e0f6790d"
server_key_iv: "f3a7acc950e2fa542e6788249b799e7c"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.795004 22865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "8df78697cde44e61b4a566117fa5af5b"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "cac42d1452010696444e4133e0f6790d"
server_key_iv: "f3a7acc950e2fa542e6788249b799e7c"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.798653 22865 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:40:27.801048 22882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.802388 22865 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:27.802527 22865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "8df78697cde44e61b4a566117fa5af5b"
format_stamp: "Formatted at 2026-08-11 04:40:27 on dist-test-slave-0kls"
server_key: "cac42d1452010696444e4133e0f6790d"
server_key_iv: "f3a7acc950e2fa542e6788249b799e7c"
server_key_version: "encryptionkey@0"
I20260811 04:40:27.802631 22865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:27.811632 22865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:27.814795 22865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:27.814990 22865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:27.815568 22865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:27.816342 22865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:27.816411 22865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.816478 22865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:27.816529 22865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:27.825894 22865 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:45899
I20260811 04:40:27.825999 23004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:45899 every 8 connection(s)
I20260811 04:40:27.826856 22865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:27.834117 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 22865
I20260811 04:40:27.834252 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:27.834650 3002 external_mini_cluster.cc:1428] Setting key e0ee073e782b2cbc6e646b19cadc5327
Aug 11 04:40:27 dist-test-slave-0kls krb5kdc[22459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423227, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:27.840320 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.828673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:54769 (local address 127.2.238.190:34483)
0811 04:40:27.828815 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.828818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.829669 (+ 851us) server_negotiation.cc:407] Connection header received
0811 04:40:27.830659 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.830662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.830721 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.830841 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:27.831978 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.832475 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.833539 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.833776 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.833927 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.836888 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:27.836892 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:27.836894 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:27.836924 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:27.838284 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.838792 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.838796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.838798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.838884 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.839247 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.839251 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.839253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.839439 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:27.839546 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.840035 (+ 489us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.840156 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 04:40:27.841107 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.829030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34483 (local address 127.2.238.131:54769)
0811 04:40:27.829505 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.829546 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.830424 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.831010 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.831017 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.831378 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:27.831839 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.831852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.832605 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.832608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.833420 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.833429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.833973 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.833976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.834153 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.834941 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:27.834958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:27.836713 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:27.838400 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.838406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.838421 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.838682 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.838985 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:27.838988 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:27.838990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:27.839139 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:27.839545 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:27.839552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:27.839800 (+ 248us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.839993 (+ 193us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.840325 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":106,"threads_started":1}
I20260811 04:40:27.842182 23005 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34483
I20260811 04:40:27.842417 23005 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:27.842890 23005 heartbeater.cc:511] Master 127.2.238.190:34483 requested a full tablet report, sending...
I20260811 04:40:27.843881 22507 ts_manager.cc:194] Registered new tserver with Master: 8df78697cde44e61b4a566117fa5af5b (127.2.238.131:45899)
I20260811 04:40:27.844444 22507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:54769
I20260811 04:40:27.848491 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:27.857725 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.850504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57388 (local address 127.2.238.190:34483)
0811 04:40:27.850623 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.850626 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.850858 (+ 232us) server_negotiation.cc:407] Connection header received
0811 04:40:27.850951 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.850953 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.850988 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.851054 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:27.851798 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.852282 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.853083 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.853293 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.853366 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.854955 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:27.854958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:27.854960 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:27.854980 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:27.856218 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.856661 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.856663 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.856664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.856733 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:27.857017 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:27.857019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:27.857020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:27.857218 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:27.857320 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.857526 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.857614 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:40:27.860486 22507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:27.863538 22507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:27.877701 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.872016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41536 (local address 127.2.238.131:45899)
0811 04:40:27.872506 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.872519 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.872545 (+ 26us) server_negotiation.cc:407] Connection header received
0811 04:40:27.872695 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.872704 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.872925 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.873023 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:27.873572 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.874724 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.875914 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.876758 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.876872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.877030 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.877442 (+ 412us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.877531 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":183,"threads_started":1}
I20260811 04:40:27.877691 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.871863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45899 (local address 127.0.0.1:41536)
0811 04:40:27.872328 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.872377 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.872506 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.873253 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.873259 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.873282 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:27.873415 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.873421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.874846 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.874848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.875715 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.875723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.877099 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.877102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.877225 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.877270 (+ 45us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.877394 (+ 124us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.877511 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":68,"threads_started":1}
I20260811 04:40:27.878468 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.872251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:33751 (local address 127.0.0.1:47158)
0811 04:40:27.872690 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.872704 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.872776 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.873993 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.873996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.874015 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:27.874126 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.874133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.875809 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.875812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.877052 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.877061 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.878160 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.878165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.878253 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.878276 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.878325 (+ 49us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.878374 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":65,"thread_start_us":85,"threads_started":1}
I20260811 04:40:27.879124 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.872667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47158 (local address 127.2.238.130:33751)
0811 04:40:27.873138 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.873239 (+ 101us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.873289 (+ 50us) server_negotiation.cc:407] Connection header received
0811 04:40:27.873379 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.873385 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.873822 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.874296 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:27.874476 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.875648 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.877173 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.877999 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.878658 (+ 659us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.878804 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.878859 (+ 55us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.878923 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":105,"threads_started":1}
I20260811 04:40:27.879410 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.872954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.2.238.129:33817)
0811 04:40:27.873286 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.873297 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.873880 (+ 583us) server_negotiation.cc:407] Connection header received
0811 04:40:27.873972 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.873978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.874191 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.874340 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:27.874542 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.875683 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.877229 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.878201 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.878316 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.878510 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.879124 (+ 614us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.879240 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":104,"threads_started":1}
I20260811 04:40:27.879769 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.872546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33817 (local address 127.0.0.1:44836)
0811 04:40:27.873594 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.873611 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.873728 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.874344 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.874348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.874366 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:27.874443 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.874449 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.875856 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.875884 (+ 28us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.877083 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.877092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.878577 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.878581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.878670 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.878696 (+ 26us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.878787 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.878887 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":980,"mutex_wait_us":27,"thread_start_us":84,"threads_started":1}
I20260811 04:40:27.881400 22932 tablet_service.cc:1467] Processing CreateTablet for tablet 54187df0891245caa248a6232138ecc0 (DEFAULT_TABLE table=test-table [id=3537d1a8d97d4a7b826d9d0fc930ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:27.882063 22788 tablet_service.cc:1467] Processing CreateTablet for tablet 54187df0891245caa248a6232138ecc0 (DEFAULT_TABLE table=test-table [id=3537d1a8d97d4a7b826d9d0fc930ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:27.882659 22932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54187df0891245caa248a6232138ecc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:27.882539 22641 tablet_service.cc:1467] Processing CreateTablet for tablet 54187df0891245caa248a6232138ecc0 (DEFAULT_TABLE table=test-table [id=3537d1a8d97d4a7b826d9d0fc930ea22]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:27.883095 22788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54187df0891245caa248a6232138ecc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:27.883317 22641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54187df0891245caa248a6232138ecc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:27.890386 23024 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Bootstrap starting.
I20260811 04:40:27.890550 23026 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Bootstrap starting.
I20260811 04:40:27.892226 23026 tablet_bootstrap.cc:654] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:27.892232 23024 tablet_bootstrap.cc:654] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:27.893175 23026 log.cc:824] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:27.893193 23024 log.cc:824] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:27.893843 23025 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: Bootstrap starting.
I20260811 04:40:27.895018 23026 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: No bootstrap required, opened a new log
I20260811 04:40:27.895151 23024 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: No bootstrap required, opened a new log
I20260811 04:40:27.895262 23026 ts_tablet_manager.cc:1397] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 04:40:27.895339 23024 ts_tablet_manager.cc:1397] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260811 04:40:27.896278 23025 tablet_bootstrap.cc:654] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:27.897035 23025 log.cc:824] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:27.898711 23026 raft_consensus.cc:348] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.898974 23026 raft_consensus.cc:374] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:27.898988 23025 tablet_bootstrap.cc:492] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: No bootstrap required, opened a new log
I20260811 04:40:27.899088 23026 raft_consensus.cc:729] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df78697cde44e61b4a566117fa5af5b, State: Initialized, Role: FOLLOWER
I20260811 04:40:27.898974 23024 raft_consensus.cc:348] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.899199 23025 ts_tablet_manager.cc:1397] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:40:27.899261 23024 raft_consensus.cc:374] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:27.899348 23024 raft_consensus.cc:729] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c71eeeb1824dfea6073a133f4b5a60, State: Initialized, Role: FOLLOWER
I20260811 04:40:27.899657 23026 consensus_queue.cc:260] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.899801 23024 consensus_queue.cc:260] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.900363 23005 heartbeater.cc:503] Master 127.2.238.190:34483 was elected leader, sending a full tablet report...
I20260811 04:40:27.900701 23026 ts_tablet_manager.cc:1428] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:40:27.901367 23024 ts_tablet_manager.cc:1428] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260811 04:40:27.901535 22717 tablet.cc:2369] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:40:27.901842 22716 heartbeater.cc:503] Master 127.2.238.190:34483 was elected leader, sending a full tablet report...
I20260811 04:40:27.902885 23025 raft_consensus.cc:348] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.903177 23025 raft_consensus.cc:374] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:27.903261 23025 raft_consensus.cc:729] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5deebf666c28436a8cee28303cded602, State: Initialized, Role: FOLLOWER
I20260811 04:40:27.903653 23025 consensus_queue.cc:260] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.904522 23025 ts_tablet_manager.cc:1428] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:40:27.904497 22860 heartbeater.cc:503] Master 127.2.238.190:34483 was elected leader, sending a full tablet report...
I20260811 04:40:27.968290 23032 raft_consensus.cc:482] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:27.968533 23032 raft_consensus.cc:504] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.969831 23032 leader_election.cc:288] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df78697cde44e61b4a566117fa5af5b (127.2.238.131:45899), b4c71eeeb1824dfea6073a133f4b5a60 (127.2.238.129:33817)
I20260811 04:40:27.974205 22862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.970030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45899 (local address 127.2.238.130:50831)
0811 04:40:27.970155 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.970167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.970252 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.970625 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.970628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.970649 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:27.970770 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.970779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.971734 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.971736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.972873 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.972883 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.973711 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.973843 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.973894 (+ 51us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.973970 (+ 76us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.974050 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260811 04:40:27.974236 23017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.970188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:50831 (local address 127.2.238.131:45899)
0811 04:40:27.970328 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.970332 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.970348 (+ 16us) server_negotiation.cc:407] Connection header received
0811 04:40:27.970402 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.970405 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.970449 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.970525 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:27.970894 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.971640 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973002 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.973597 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973716 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.973806 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.974002 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.974091 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260811 04:40:27.974262 23033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.970298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33817 (local address 127.2.238.130:53533)
0811 04:40:27.970632 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:27.970643 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:27.970725 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:27.970973 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:27.970975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:27.970989 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:27.971092 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.971096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.971932 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.971935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.973026 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:27.973033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.973839 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:27.973972 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.973991 (+ 19us) client_negotiation.cc:769] Sending connection context
0811 04:40:27.974052 (+ 61us) client_negotiation.cc:240] Negotiation successful
0811 04:40:27.974149 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":19,"thread_start_us":147,"threads_started":1}
I20260811 04:40:27.974277 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.970483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:53533 (local address 127.2.238.129:33817)
0811 04:40:27.970598 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.970603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.970721 (+ 118us) server_negotiation.cc:407] Connection header received
0811 04:40:27.970802 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.970804 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.970880 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.970960 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:27.971217 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.971847 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973139 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.973722 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:27.973857 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:27.973891 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:27.974090 (+ 199us) server_negotiation.cc:299] Negotiation successful
0811 04:40:27.974173 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:40:27.974929 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54187df0891245caa248a6232138ecc0" candidate_uuid: "5deebf666c28436a8cee28303cded602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" is_pre_election: true
I20260811 04:40:27.974953 22955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54187df0891245caa248a6232138ecc0" candidate_uuid: "5deebf666c28436a8cee28303cded602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df78697cde44e61b4a566117fa5af5b" is_pre_election: true
I20260811 04:40:27.975227 22661 raft_consensus.cc:2454] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deebf666c28436a8cee28303cded602 in term 0.
I20260811 04:40:27.975265 22955 raft_consensus.cc:2454] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5deebf666c28436a8cee28303cded602 in term 0.
I20260811 04:40:27.975721 22738 leader_election.cc:302] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deebf666c28436a8cee28303cded602, 8df78697cde44e61b4a566117fa5af5b; no voters:
I20260811 04:40:27.976008 23032 raft_consensus.cc:2781] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:27.976082 23032 raft_consensus.cc:482] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:27.976111 23032 raft_consensus.cc:3037] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:27.977154 23032 raft_consensus.cc:504] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.977484 23032 leader_election.cc:288] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [CANDIDATE]: Term 1 election: Requested vote from peers 8df78697cde44e61b4a566117fa5af5b (127.2.238.131:45899), b4c71eeeb1824dfea6073a133f4b5a60 (127.2.238.129:33817)
I20260811 04:40:27.977857 22955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54187df0891245caa248a6232138ecc0" candidate_uuid: "5deebf666c28436a8cee28303cded602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df78697cde44e61b4a566117fa5af5b"
I20260811 04:40:27.977926 22661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54187df0891245caa248a6232138ecc0" candidate_uuid: "5deebf666c28436a8cee28303cded602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4c71eeeb1824dfea6073a133f4b5a60"
I20260811 04:40:27.977972 22955 raft_consensus.cc:3037] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:27.978065 22661 raft_consensus.cc:3037] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:27.979055 22955 raft_consensus.cc:2454] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deebf666c28436a8cee28303cded602 in term 1.
I20260811 04:40:27.979192 22661 raft_consensus.cc:2454] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5deebf666c28436a8cee28303cded602 in term 1.
I20260811 04:40:27.979347 22738 leader_election.cc:302] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5deebf666c28436a8cee28303cded602, 8df78697cde44e61b4a566117fa5af5b; no voters:
I20260811 04:40:27.979547 23032 raft_consensus.cc:2781] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:27.979765 23032 raft_consensus.cc:686] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 1 LEADER]: Becoming Leader. State: Replica: 5deebf666c28436a8cee28303cded602, State: Running, Role: LEADER
I20260811 04:40:27.980176 23032 consensus_queue.cc:237] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } }
I20260811 04:40:27.983302 22507 catalog_manager.cc:5495] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5deebf666c28436a8cee28303cded602 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "5deebf666c28436a8cee28303cded602" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5deebf666c28436a8cee28303cded602" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 33751 } health_report { overall_health: HEALTHY } } }
I20260811 04:40:28.001983 23022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:27.998279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47166 (local address 127.2.238.130:33751)
0811 04:40:27.998400 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:27.998403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:27.998415 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:40:27.998540 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:27.998542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:27.998589 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:27.998676 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:27.998900 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:27.999342 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.000280 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.000490 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.000616 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.000861 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:28.001295 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:28.001495 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.001702 (+ 207us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.001743 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 04:40:28.009613 22661 raft_consensus.cc:1261] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60 [term 1 FOLLOWER]: Refusing update from remote peer 5deebf666c28436a8cee28303cded602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:28.009775 22955 raft_consensus.cc:1261] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 1 FOLLOWER]: Refusing update from remote peer 5deebf666c28436a8cee28303cded602: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:28.010492 23032 consensus_queue.cc:1035] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df78697cde44e61b4a566117fa5af5b" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:28.010739 23034 consensus_queue.cc:1035] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4c71eeeb1824dfea6073a133f4b5a60" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:28.018349 23038 mvcc.cc:205] Tried to move back new op lower bound from 7317189541918072832 to 7317189541809795072. Current Snapshot: MvccSnapshot[applied={T|T < 7317189541918072832}]
I20260811 04:40:28.019829 23039 mvcc.cc:205] Tried to move back new op lower bound from 7317189541918072832 to 7317189541809795072. Current Snapshot: MvccSnapshot[applied={T|T < 7317189541918072832}]
I20260811 04:40:28.026809 22507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:28.026950 22507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:28.029172 22507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 06dd2dd715a847e09bcb16f187b18e87: Sending DeleteTablet for 3 replicas of tablet 54187df0891245caa248a6232138ecc0
I20260811 04:40:28.030007 22788 tablet_service.cc:1514] Processing DeleteTablet for tablet 54187df0891245caa248a6232138ecc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:28 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:47158
I20260811 04:40:28.030262 22932 tablet_service.cc:1514] Processing DeleteTablet for tablet 54187df0891245caa248a6232138ecc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:28 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:41536
I20260811 04:40:28.030287 22641 tablet_service.cc:1514] Processing DeleteTablet for tablet 54187df0891245caa248a6232138ecc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:28 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:44836
I20260811 04:40:28.030756 23048 tablet_replica.cc:330] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: stopping tablet replica
I20260811 04:40:28.030956 23050 tablet_replica.cc:330] T 54187df0891245caa248a6232138ecc0 P b4c71eeeb1824dfea6073a133f4b5a60: stopping tablet replica
I20260811 04:40:28.031095 23048 raft_consensus.cc:2229] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:28.031246 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22574
I20260811 04:40:28.031384 23048 raft_consensus.cc:2258] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:28.033434 23048 ts_tablet_manager.cc:1905] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:28.036973 23048 ts_tablet_manager.cc:1918] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:28.037094 23048 log.cc:1196] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/54187df0891245caa248a6232138ecc0
I20260811 04:40:28.037449 23048 ts_tablet_manager.cc:1939] T 54187df0891245caa248a6232138ecc0 P 8df78697cde44e61b4a566117fa5af5b: Deleting consensus metadata
I20260811 04:40:28.038875 22490 catalog_manager.cc:4841] TS 8df78697cde44e61b4a566117fa5af5b (127.2.238.131:45899): tablet 54187df0891245caa248a6232138ecc0 (table test-table [id=3537d1a8d97d4a7b826d9d0fc930ea22]) successfully deleted
W20260811 04:40:28.042409 22490 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:33817 (error 108)
I20260811 04:40:28.042606 23047 tablet_replica.cc:330] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602: stopping tablet replica
I20260811 04:40:28.042912 23047 raft_consensus.cc:2229] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:28.043272 23047 raft_consensus.cc:2258] T 54187df0891245caa248a6232138ecc0 P 5deebf666c28436a8cee28303cded602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:28.043567 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.043249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33817 (local address 127.0.0.1:44846)
0811 04:40:28.043383 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.043460 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260811 04:40:28.043642 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22721
W20260811 04:40:28.043716 22490 catalog_manager.cc:4584] TS b4c71eeeb1824dfea6073a133f4b5a60 (127.2.238.129:33817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54187df0891245caa248a6232138ecc0: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33817: connect: Connection refused (error 111)
I20260811 04:40:28.051400 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22865
I20260811 04:40:28.051708 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.051441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:33751 (local address 127.0.0.1:47168)
0811 04:40:28.051569 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.051625 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.130:33751: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260811 04:40:28.059587 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 22473
2026-08-11T04:40:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2284 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-11T04:40:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:28Z Disabled control of system clock
I20260811 04:40:28.092494 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:35623
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:35623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:28.200145 23055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:28.200388 23055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:28.200423 23055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:28.204107 23055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:28.204155 23055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:28.204188 23055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:28.204205 23055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:28.204260 23055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:28.208882 23055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:35623
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:35623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:28.209931 23055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:28.211212 23055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:28.218922 23063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:28.218941 23060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.219048 23055 server_base.cc:1034] running on GCE node
W20260811 04:40:28.219226 23061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.219800 23055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:28.220840 23055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:28.222007 23055 hybrid_clock.cc:648] HybridClock initialized: now 1786423228221975 us; error 41 us; skew 500 ppm
I20260811 04:40:28.228178 23055 webserver.cc:460] Webserver started at http://127.2.238.190:39457/ using document root <none> and password file <none>
I20260811 04:40:28.228762 23055 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:28.228833 23055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:28.229066 23055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:28.231249 23055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "8f9782fc2766d9a25809c7dd986a2233"
server_key_iv: "67644c6a3a108ca89aefc9bc6eef344f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.231814 23055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "8f9782fc2766d9a25809c7dd986a2233"
server_key_iv: "67644c6a3a108ca89aefc9bc6eef344f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.235510 23055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260811 04:40:28.237687 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.238832 23055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:40:28.238962 23055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "8f9782fc2766d9a25809c7dd986a2233"
server_key_iv: "67644c6a3a108ca89aefc9bc6eef344f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.239063 23055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:28.249194 23055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:28.249873 23055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:28.250057 23055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:28.256910 23055 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:35623
I20260811 04:40:28.256929 23127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:35623 every 8 connection(s)
I20260811 04:40:28.257774 23055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:28.258070 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23055
I20260811 04:40:28.258209 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:28.258556 3002 external_mini_cluster.cc:1428] Setting key a5bda8d60d4cf3887223edf7b2400819
I20260811 04:40:28.261227 23128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:28.266911 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: Bootstrap starting.
I20260811 04:40:28.267530 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.259865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.2.238.190:35623)
0811 04:40:28.260518 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.260539 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.260583 (+ 44us) server_negotiation.cc:407] Connection header received
0811 04:40:28.261247 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.261261 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.261652 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.262005 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:28.262424 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.263316 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.264117 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.264598 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.264792 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.265753 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:28.265758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:28.265770 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:28.266201 (+ 431us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:28.266254 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:28.266275 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:28.266385 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.266540 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.266810 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":135,"threads_started":1}
I20260811 04:40:28.269383 23128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:28.270264 23128 log.cc:824] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:28.272185 23128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: No bootstrap required, opened a new log
I20260811 04:40:28.275651 23128 raft_consensus.cc:348] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } }
I20260811 04:40:28.275882 23128 raft_consensus.cc:374] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:28.275975 23128 raft_consensus.cc:729] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de5b3a4b8f304dc799ab2dc873ac2bcd, State: Initialized, Role: FOLLOWER
I20260811 04:40:28.276425 23128 consensus_queue.cc:260] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } }
I20260811 04:40:28.276575 23128 raft_consensus.cc:388] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:28.276669 23128 raft_consensus.cc:482] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:28.276779 23128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:28.277640 23128 raft_consensus.cc:504] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } }
I20260811 04:40:28.277978 23128 leader_election.cc:302] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de5b3a4b8f304dc799ab2dc873ac2bcd; no voters:
I20260811 04:40:28.278262 23128 leader_election.cc:288] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:28.278375 23133 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:28.278606 23133 raft_consensus.cc:686] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [term 1 LEADER]: Becoming Leader. State: Replica: de5b3a4b8f304dc799ab2dc873ac2bcd, State: Running, Role: LEADER
I20260811 04:40:28.278874 23133 consensus_queue.cc:237] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } }
I20260811 04:40:28.279476 23128 sys_catalog.cc:564] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:28.280578 23135 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader de5b3a4b8f304dc799ab2dc873ac2bcd. Latest consensus state: current_term: 1 leader_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } } }
I20260811 04:40:28.280706 23135 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:28.280992 23134 sys_catalog.cc:455] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de5b3a4b8f304dc799ab2dc873ac2bcd" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 35623 } } }
I20260811 04:40:28.281080 23134 sys_catalog.cc:458] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:28.281211 23142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:28.283878 23142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:28.289525 23142 catalog_manager.cc:1347] Generated new cluster ID: b69d0e6f01724a6384e784938ca41707
I20260811 04:40:28.289590 23142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:28.294978 23142 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:28.295814 23142 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:28.300921 23142 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: Generated new TSK 0
I20260811 04:40:28.301621 23142 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:28.306269 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--builtin_ntp_servers=127.2.238.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:28.441267 23153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:28.441563 23153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:28.441663 23153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:28.445959 23153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:28.446031 23153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:28.446111 23153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:28.450832 23153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:28.452003 23153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:28.453460 23153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:28.463409 23161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:28.463474 23158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.463547 23153 server_base.cc:1034] running on GCE node
W20260811 04:40:28.463409 23159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.464226 23153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:28.464880 23153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:28.466076 23153 hybrid_clock.cc:648] HybridClock initialized: now 1786423228466054 us; error 43 us; skew 500 ppm
I20260811 04:40:28.467891 23153 webserver.cc:460] Webserver started at http://127.2.238.129:33583/ using document root <none> and password file <none>
I20260811 04:40:28.468467 23153 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:28.468559 23153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:28.468799 23153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:28.470901 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "3876a97eb62f4e1f8c66b3e140142d70"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "3ea677f461d64b3e776fb69c2c7f9afa"
server_key_iv: "2171cc7cb8c5454c94c42fc4c8ba9b9f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.471472 23153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "3876a97eb62f4e1f8c66b3e140142d70"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "3ea677f461d64b3e776fb69c2c7f9afa"
server_key_iv: "2171cc7cb8c5454c94c42fc4c8ba9b9f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.475214 23153 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:28.477914 23167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.479101 23153 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:28.479259 23153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "3876a97eb62f4e1f8c66b3e140142d70"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "3ea677f461d64b3e776fb69c2c7f9afa"
server_key_iv: "2171cc7cb8c5454c94c42fc4c8ba9b9f"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.479377 23153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:28.496064 23153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:28.496789 23153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:28.496986 23153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:28.497582 23153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:28.498528 23153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:28.498596 23153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.498674 23153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:28.498718 23153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.507831 23153 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:33359
I20260811 04:40:28.507856 23292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:33359 every 8 connection(s)
I20260811 04:40:28.508715 23153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:28.515079 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23153
I20260811 04:40:28.515232 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:40:28.515548 3002 external_mini_cluster.cc:1428] Setting key 148c5dde4bfc61145d459cb60655b0d0
I20260811 04:40:28.517765 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--builtin_ntp_servers=127.2.238.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:28.517884 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.510553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:38675 (local address 127.2.238.190:35623)
0811 04:40:28.510694 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.510698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.511696 (+ 998us) server_negotiation.cc:407] Connection header received
0811 04:40:28.512983 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.512986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.513033 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.513116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:28.513973 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.514514 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.515754 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.516031 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.516143 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.517326 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:28.517330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:28.517332 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:28.517344 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:28.517367 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:28.517376 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:28.517447 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.517583 (+ 136us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.517688 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:40:28.518695 23295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.510959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35623 (local address 127.2.238.129:38675)
0811 04:40:28.511574 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.511610 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.512807 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.513286 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.513292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.513369 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:28.513862 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.513875 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.514621 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.514623 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.515630 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.515649 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.516198 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.516201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.516381 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.517051 (+ 670us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:28.517053 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:28.517105 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:28.517190 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:28.517454 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:28.517462 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:28.517466 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.517553 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.517778 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":114,"threads_started":1}
I20260811 04:40:28.520222 23293 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35623
I20260811 04:40:28.520504 23293 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:28.521123 23293 heartbeater.cc:511] Master 127.2.238.190:35623 requested a full tablet report, sending...
I20260811 04:40:28.522838 23087 ts_manager.cc:194] Registered new tserver with Master: 3876a97eb62f4e1f8c66b3e140142d70 (127.2.238.129:33359)
I20260811 04:40:28.523964 23087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.129:38675
W20260811 04:40:28.624190 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:28.624446 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:28.624513 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:28.628203 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:28.628270 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:28.628348 23296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:28.632759 23296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:28.633735 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:28.635131 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:28.644556 23302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:28.644635 23301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:28.644583 23304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.644862 23296 server_base.cc:1034] running on GCE node
I20260811 04:40:28.645293 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:28.645983 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:28.647167 23296 hybrid_clock.cc:648] HybridClock initialized: now 1786423228647155 us; error 22 us; skew 500 ppm
I20260811 04:40:28.648877 23296 webserver.cc:460] Webserver started at http://127.2.238.130:38209/ using document root <none> and password file <none>
I20260811 04:40:28.649451 23296 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:28.649540 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:28.649757 23296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:28.651770 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "84b97b1792f94a88bdf55afcf387b502"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "a4af6eecc1e19d01611f1547e131c0fd"
server_key_iv: "323c0610e7581c7ce6ddd7deb26e3c7d"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.652307 23296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "84b97b1792f94a88bdf55afcf387b502"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "a4af6eecc1e19d01611f1547e131c0fd"
server_key_iv: "323c0610e7581c7ce6ddd7deb26e3c7d"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.655949 23296 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:40:28.658286 23310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.659427 23296 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:28.659552 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "84b97b1792f94a88bdf55afcf387b502"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "a4af6eecc1e19d01611f1547e131c0fd"
server_key_iv: "323c0610e7581c7ce6ddd7deb26e3c7d"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.659667 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:28.695345 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:28.695992 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:28.696246 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:28.696853 23296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:28.697856 23296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:28.697926 23296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.697996 23296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:28.698053 23296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.706516 23296 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:34575
I20260811 04:40:28.706553 23434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:34575 every 8 connection(s)
I20260811 04:40:28.707370 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:28.715550 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23296
I20260811 04:40:28.715643 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:28.716005 3002 external_mini_cluster.cc:1428] Setting key 8e8544c6ebcbb72b4b353f6dcb1bead7
I20260811 04:40:28.716300 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.709016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:60591 (local address 127.2.238.190:35623)
0811 04:40:28.709184 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.709188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.710219 (+ 1031us) server_negotiation.cc:407] Connection header received
0811 04:40:28.711525 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.711528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.711587 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.711683 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:28.712591 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.713111 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.714305 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.714528 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.714642 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.715775 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:28.715779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:28.715782 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:28.715797 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:28.715822 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:28.715832 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:28.715931 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.716079 (+ 148us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.716145 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260811 04:40:28.717211 23437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.709410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35623 (local address 127.2.238.130:60591)
0811 04:40:28.710069 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.710109 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.711322 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.711899 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.711910 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.712007 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:28.712456 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.712468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.713246 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.713249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.714169 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.714180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.714688 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.714692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.714849 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.715485 (+ 636us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:28.715487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:28.715521 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:28.715576 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:28.715941 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:28.715948 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:28.715953 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.716047 (+ 94us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.716318 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":115,"threads_started":1}
I20260811 04:40:28.718153 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--builtin_ntp_servers=127.2.238.148:41981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:28.718403 23435 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35623
I20260811 04:40:28.718608 23435 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:28.719187 23435 heartbeater.cc:511] Master 127.2.238.190:35623 requested a full tablet report, sending...
I20260811 04:40:28.720263 23087 ts_manager.cc:194] Registered new tserver with Master: 84b97b1792f94a88bdf55afcf387b502 (127.2.238.130:34575)
I20260811 04:40:28.720817 23087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.130:60591
W20260811 04:40:28.830669 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:28.830919 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:28.830974 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:28.834792 23438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:28.834834 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:28.834920 23438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:28.839385 23438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:41981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:35623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:28.840377 23438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:28.841694 23438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:28.851069 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:28.851109 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.851231 23438 server_base.cc:1034] running on GCE node
W20260811 04:40:28.851408 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:28.851873 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:28.852524 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:28.853725 23438 hybrid_clock.cc:648] HybridClock initialized: now 1786423228853695 us; error 49 us; skew 500 ppm
I20260811 04:40:28.855739 23438 webserver.cc:460] Webserver started at http://127.2.238.131:44449/ using document root <none> and password file <none>
I20260811 04:40:28.856289 23438 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:28.856367 23438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:28.856593 23438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:28.858678 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "77035b3f7900100d4b5b58e7c215cb6c"
server_key_iv: "5c507c0c49f7eb9dcf9af40f62531212"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.859212 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "77035b3f7900100d4b5b58e7c215cb6c"
server_key_iv: "5c507c0c49f7eb9dcf9af40f62531212"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.862812 23438 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:40:28.865267 23454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.866509 23438 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:28.866648 23438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd"
format_stamp: "Formatted at 2026-08-11 04:40:28 on dist-test-slave-0kls"
server_key: "77035b3f7900100d4b5b58e7c215cb6c"
server_key_iv: "5c507c0c49f7eb9dcf9af40f62531212"
server_key_version: "encryptionkey@0"
I20260811 04:40:28.866756 23438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:28.894941 23438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:28.895654 23438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:28.895879 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:28.896492 23438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:28.897464 23438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:28.897538 23438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.897609 23438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:28.897665 23438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:28.906574 23438 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:44863
I20260811 04:40:28.906597 23580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:44863 every 8 connection(s)
I20260811 04:40:28.907442 23438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:28.915756 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23438
I20260811 04:40:28.915879 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:28.916244 3002 external_mini_cluster.cc:1428] Setting key 5d297115532a3a27617172cde83fe146
I20260811 04:40:28.916441 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.909254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:51919 (local address 127.2.238.190:35623)
0811 04:40:28.909404 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.909409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.910496 (+ 1087us) server_negotiation.cc:407] Connection header received
0811 04:40:28.911815 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.911818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.911866 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.911940 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:28.912793 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.913323 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.914496 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.914716 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.914799 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.915937 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:28.915940 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:28.915943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:28.915958 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:28.915980 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:28.915990 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:28.916080 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.916230 (+ 150us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.916297 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260811 04:40:28.917498 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.909577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:35623 (local address 127.2.238.131:51919)
0811 04:40:28.910324 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.910364 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.911632 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.912101 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.912109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.912191 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:28.912665 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.912678 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.913460 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.913463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.914379 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.914389 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.914852 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.914855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.914991 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.915641 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:28.915642 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:28.915687 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:28.915748 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:28.916080 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:28.916087 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:28.916092 (+ 5us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.916194 (+ 102us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.916457 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":171,"threads_started":1}
I20260811 04:40:28.918684 23582 heartbeater.cc:347] Connected to a master server at 127.2.238.190:35623
I20260811 04:40:28.918875 23582 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:28.919307 23582 heartbeater.cc:511] Master 127.2.238.190:35623 requested a full tablet report, sending...
I20260811 04:40:28.920252 23087 ts_manager.cc:194] Registered new tserver with Master: c6c619cf4dcb40d38bcc6bb630ec32cd (127.2.238.131:44863)
I20260811 04:40:28.920758 23087 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.131:51919
I20260811 04:40:28.930224 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:28.935724 23131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.932073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.2.238.190:35623)
0811 04:40:28.932214 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.932221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.932366 (+ 145us) server_negotiation.cc:407] Connection header received
0811 04:40:28.932460 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.932462 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.932501 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.932564 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:28.932807 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.933288 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.934089 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.934297 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.934373 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.935242 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:28.935252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:28.935254 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:28.935264 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:28.935280 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:28.935287 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:28.935376 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.935504 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.935559 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 04:40:28.938433 23087 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:28.941326 23087 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:28.955955 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.949853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56680 (local address 127.2.238.131:44863)
0811 04:40:28.950258 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.950270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.950753 (+ 483us) server_negotiation.cc:407] Connection header received
0811 04:40:28.950835 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.950841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.951038 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.951219 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:28.951682 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.952878 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.954576 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.955307 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.955424 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.955623 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.955677 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.955770 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":172,"threads_started":1}
I20260811 04:40:28.955922 23594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.949651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:44863 (local address 127.0.0.1:56680)
0811 04:40:28.950412 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.950460 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.950602 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.951390 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.951396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.951420 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:28.951563 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.951569 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.953004 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.953007 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.954451 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.954461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.955428 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.955430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.955507 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.955548 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.955619 (+ 71us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.955758 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":143,"thread_start_us":71,"threads_started":1}
I20260811 04:40:28.956453 23599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.950382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:34575 (local address 127.0.0.1:57832)
0811 04:40:28.950830 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.950844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.950940 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.951563 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.951565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.951597 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:28.951712 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.951720 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.953049 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.953052 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.954671 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.954683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.955708 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.955711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.955785 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.955805 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.955864 (+ 59us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.955924 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":185,"spinlock_wait_cycles":4608,"thread_start_us":107,"threads_started":1}
I20260811 04:40:28.957448 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.950090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.2.238.130:34575)
0811 04:40:28.950784 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.950797 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.951042 (+ 245us) server_negotiation.cc:407] Connection header received
0811 04:40:28.951131 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.951139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.951335 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.951486 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:28.951835 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.952953 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.954803 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.955572 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.955693 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.955860 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.955914 (+ 54us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.957278 (+ 1364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":86,"threads_started":1}
I20260811 04:40:28.957461 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.950574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33359 (local address 127.0.0.1:36102)
0811 04:40:28.951472 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:28.951485 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:28.951595 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:28.952068 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:28.952071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:28.952088 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:28.952169 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.952175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.953209 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.953215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.954671 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:28.954683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.957038 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.957041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:28.957162 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.957185 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:40:28.957274 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 04:40:28.957340 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"mutex_wait_us":72,"spinlock_wait_cycles":896,"thread_start_us":123,"threads_started":1}
I20260811 04:40:28.957894 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:28.951046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36102 (local address 127.2.238.129:33359)
0811 04:40:28.951486 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:28.951497 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:28.951696 (+ 199us) server_negotiation.cc:407] Connection header received
0811 04:40:28.951783 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:28.951790 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:28.951941 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:28.952088 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:28.952293 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.953081 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.956196 (+ 3115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:28.956918 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:28.957029 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:28.957202 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:28.957611 (+ 409us) server_negotiation.cc:299] Negotiation successful
0811 04:40:28.957707 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":127,"threads_started":1}
I20260811 04:40:28.959448 23510 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a (DEFAULT_TABLE table=test-table [id=9d36033fc162472285bbd5edc7f49519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:28.960076 23220 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a (DEFAULT_TABLE table=test-table [id=9d36033fc162472285bbd5edc7f49519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:28.960592 23510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:28.960213 23362 tablet_service.cc:1467] Processing CreateTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a (DEFAULT_TABLE table=test-table [id=9d36033fc162472285bbd5edc7f49519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:28.960841 23220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:28.961158 23362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:28.967082 23601 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Bootstrap starting.
I20260811 04:40:28.969425 23601 tablet_bootstrap.cc:654] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:28.970155 23603 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Bootstrap starting.
I20260811 04:40:28.970325 23601 log.cc:824] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:28.970991 23602 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Bootstrap starting.
I20260811 04:40:28.972390 23603 tablet_bootstrap.cc:654] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:28.972486 23601 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: No bootstrap required, opened a new log
I20260811 04:40:28.972694 23601 ts_tablet_manager.cc:1397] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:40:28.972992 23602 tablet_bootstrap.cc:654] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:28.973245 23603 log.cc:824] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:28.973730 23602 log.cc:824] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:28.975340 23602 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: No bootstrap required, opened a new log
I20260811 04:40:28.975340 23603 tablet_bootstrap.cc:492] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: No bootstrap required, opened a new log
I20260811 04:40:28.975544 23602 ts_tablet_manager.cc:1397] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260811 04:40:28.975544 23603 ts_tablet_manager.cc:1397] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:40:28.977066 23601 raft_consensus.cc:348] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.977348 23601 raft_consensus.cc:374] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:28.977397 23601 raft_consensus.cc:729] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3876a97eb62f4e1f8c66b3e140142d70, State: Initialized, Role: FOLLOWER
I20260811 04:40:28.977885 23601 consensus_queue.cc:260] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.978860 23601 ts_tablet_manager.cc:1428] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260811 04:40:28.979053 23602 raft_consensus.cc:348] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.979265 23293 heartbeater.cc:503] Master 127.2.238.190:35623 was elected leader, sending a full tablet report...
I20260811 04:40:28.979331 23602 raft_consensus.cc:374] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:28.979393 23602 raft_consensus.cc:729] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c619cf4dcb40d38bcc6bb630ec32cd, State: Initialized, Role: FOLLOWER
I20260811 04:40:28.979511 23603 raft_consensus.cc:348] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.979741 23603 raft_consensus.cc:374] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:28.979808 23603 raft_consensus.cc:729] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b97b1792f94a88bdf55afcf387b502, State: Initialized, Role: FOLLOWER
I20260811 04:40:28.979825 23602 consensus_queue.cc:260] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.980242 23603 consensus_queue.cc:260] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:28.980473 23582 heartbeater.cc:503] Master 127.2.238.190:35623 was elected leader, sending a full tablet report...
I20260811 04:40:28.980803 23602 ts_tablet_manager.cc:1428] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260811 04:40:28.981165 23603 ts_tablet_manager.cc:1428] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:40:28.981465 23435 heartbeater.cc:503] Master 127.2.238.190:35623 was elected leader, sending a full tablet report...
W20260811 04:40:29.009904 23294 tablet.cc:2369] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:29.158471 23583 tablet.cc:2369] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:29.208483 23436 tablet.cc:2369] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:40:29.213207 23608 raft_consensus.cc:482] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:29.213445 23608 raft_consensus.cc:504] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:29.214671 23608 leader_election.cc:288] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b97b1792f94a88bdf55afcf387b502 (127.2.238.130:34575), 3876a97eb62f4e1f8c66b3e140142d70 (127.2.238.129:33359)
I20260811 04:40:29.218452 23596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.214987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:59903 (local address 127.2.238.130:34575)
0811 04:40:29.215151 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:29.215156 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:29.215175 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:40:29.215323 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:29.215326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:29.215382 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:29.215465 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:29.215762 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.216238 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.217392 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.217924 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.218041 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:29.218090 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:29.218280 (+ 190us) server_negotiation.cc:299] Negotiation successful
0811 04:40:29.218342 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 04:40:29.218452 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.214876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:34575 (local address 127.2.238.131:59903)
0811 04:40:29.215086 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:29.215102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:29.215209 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:29.215489 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:29.215491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:29.215542 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:29.215647 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:29.215653 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.216343 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.216349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:29.217280 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:29.217289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.218041 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.218044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:29.218144 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:29.218185 (+ 41us) client_negotiation.cc:769] Sending connection context
0811 04:40:29.218263 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:40:29.218327 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":82,"spinlock_wait_cycles":41600}
I20260811 04:40:29.218595 23600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.215093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:34479 (local address 127.2.238.129:33359)
0811 04:40:29.215222 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:29.215227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:29.215492 (+ 265us) server_negotiation.cc:407] Connection header received
0811 04:40:29.215678 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:29.215681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:29.215736 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:29.215826 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:29.216116 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.216614 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.217612 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.218032 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.218161 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:29.218212 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:29.218388 (+ 176us) server_negotiation.cc:299] Negotiation successful
0811 04:40:29.218455 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:40:29.218993 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.214978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33359 (local address 127.2.238.131:34479)
0811 04:40:29.215377 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:29.215388 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:29.215558 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:29.215838 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:29.215841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:29.215857 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:29.215987 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:29.215992 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.216746 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.216750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:29.217485 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:29.217493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.218148 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.218151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:29.218263 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:29.218288 (+ 25us) client_negotiation.cc:769] Sending connection context
0811 04:40:29.218369 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 04:40:29.218431 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":24,"spinlock_wait_cycles":6912,"thread_start_us":115,"threads_started":1}
I20260811 04:40:29.219174 23385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4a5b3a31b045619ed78ae7d24c9b1a" candidate_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b97b1792f94a88bdf55afcf387b502" is_pre_election: true
I20260811 04:40:29.219489 23385 raft_consensus.cc:2454] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c619cf4dcb40d38bcc6bb630ec32cd in term 0.
I20260811 04:40:29.219659 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4a5b3a31b045619ed78ae7d24c9b1a" candidate_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3876a97eb62f4e1f8c66b3e140142d70" is_pre_election: true
I20260811 04:40:29.219950 23241 raft_consensus.cc:2454] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c619cf4dcb40d38bcc6bb630ec32cd in term 0.
I20260811 04:40:29.220093 23456 leader_election.cc:302] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84b97b1792f94a88bdf55afcf387b502, c6c619cf4dcb40d38bcc6bb630ec32cd; no voters:
I20260811 04:40:29.220383 23608 raft_consensus.cc:2781] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:29.220454 23608 raft_consensus.cc:482] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:29.220484 23608 raft_consensus.cc:3037] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:29.221477 23608 raft_consensus.cc:504] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:29.221849 23608 leader_election.cc:288] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [CANDIDATE]: Term 1 election: Requested vote from peers 84b97b1792f94a88bdf55afcf387b502 (127.2.238.130:34575), 3876a97eb62f4e1f8c66b3e140142d70 (127.2.238.129:33359)
I20260811 04:40:29.222172 23385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4a5b3a31b045619ed78ae7d24c9b1a" candidate_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b97b1792f94a88bdf55afcf387b502"
I20260811 04:40:29.222208 23241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a4a5b3a31b045619ed78ae7d24c9b1a" candidate_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3876a97eb62f4e1f8c66b3e140142d70"
I20260811 04:40:29.222291 23385 raft_consensus.cc:3037] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:29.222348 23241 raft_consensus.cc:3037] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:29.223206 23385 raft_consensus.cc:2454] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c619cf4dcb40d38bcc6bb630ec32cd in term 1.
I20260811 04:40:29.223351 23241 raft_consensus.cc:2454] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c619cf4dcb40d38bcc6bb630ec32cd in term 1.
I20260811 04:40:29.223558 23456 leader_election.cc:302] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84b97b1792f94a88bdf55afcf387b502, c6c619cf4dcb40d38bcc6bb630ec32cd; no voters:
I20260811 04:40:29.223706 23608 raft_consensus.cc:2781] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:29.223870 23608 raft_consensus.cc:686] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 1 LEADER]: Becoming Leader. State: Replica: c6c619cf4dcb40d38bcc6bb630ec32cd, State: Running, Role: LEADER
I20260811 04:40:29.224195 23608 consensus_queue.cc:237] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } }
I20260811 04:40:29.227289 23085 catalog_manager.cc:5495] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd reported cstate change: term changed from 0 to 1, leader changed from <none> to c6c619cf4dcb40d38bcc6bb630ec32cd (127.2.238.131). New cstate: current_term: 1 leader_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c619cf4dcb40d38bcc6bb630ec32cd" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 44863 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 } health_report { overall_health: UNKNOWN } } }
I20260811 04:40:29.247401 23595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.244054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.2.238.131:44863)
0811 04:40:29.244172 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:29.244175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:29.244206 (+ 31us) server_negotiation.cc:407] Connection header received
0811 04:40:29.244322 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:29.244324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:29.244372 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:29.244455 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:29.244726 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.245196 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.246195 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:29.246396 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:29.246469 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:29.246724 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:29.247088 (+ 364us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:29.247198 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:29.247279 (+ 81us) server_negotiation.cc:299] Negotiation successful
0811 04:40:29.247307 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 04:40:29.253110 23385 raft_consensus.cc:1261] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 1 FOLLOWER]: Refusing update from remote peer c6c619cf4dcb40d38bcc6bb630ec32cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:29.253110 23241 raft_consensus.cc:1261] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 1 FOLLOWER]: Refusing update from remote peer c6c619cf4dcb40d38bcc6bb630ec32cd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:29.253947 23608 consensus_queue.cc:1035] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3876a97eb62f4e1f8c66b3e140142d70" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 33359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:29.254158 23612 consensus_queue.cc:1035] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b97b1792f94a88bdf55afcf387b502" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 34575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:29.263077 23618 mvcc.cc:205] Tried to move back new op lower bound from 7317189547012870144 to 7317189546904842240. Current Snapshot: MvccSnapshot[applied={T|T < 7317189547012870144}]
I20260811 04:40:29.263125 23619 mvcc.cc:205] Tried to move back new op lower bound from 7317189547012870144 to 7317189546904842240. Current Snapshot: MvccSnapshot[applied={T|T < 7317189547012870144}]
I20260811 04:40:29.274762 23084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:29.274951 23084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:29.277475 23084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de5b3a4b8f304dc799ab2dc873ac2bcd: Sending DeleteTablet for 3 replicas of tablet 6a4a5b3a31b045619ed78ae7d24c9b1a
I20260811 04:40:29.278291 23510 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:29 UTC) from {username='slave'} at 127.0.0.1:56680
I20260811 04:40:29.278399 23362 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:29 UTC) from {username='slave'} at 127.0.0.1:57832
I20260811 04:40:29.278545 23220 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:29 UTC) from {username='slave'} at 127.0.0.1:36102
I20260811 04:40:29.278765 23627 tablet_replica.cc:330] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: stopping tablet replica
I20260811 04:40:29.279060 23627 raft_consensus.cc:2229] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:29.279208 23629 tablet_replica.cc:330] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70: stopping tablet replica
I20260811 04:40:29.279016 23628 tablet_replica.cc:330] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: stopping tablet replica
I20260811 04:40:29.279475 23629 raft_consensus.cc:2229] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:29.279709 23628 raft_consensus.cc:2229] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:29.279860 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23153
I20260811 04:40:29.279881 23629 raft_consensus.cc:2258] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 3876a97eb62f4e1f8c66b3e140142d70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:29.282480 23627 raft_consensus.cc:2258] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:29.282851 23628 raft_consensus.cc:2258] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:29.284278 23627 ts_tablet_manager.cc:1905] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:29.284679 23628 ts_tablet_manager.cc:1905] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:29.287036 23627 ts_tablet_manager.cc:1918] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:29.287128 23627 log.cc:1196] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/6a4a5b3a31b045619ed78ae7d24c9b1a
I20260811 04:40:29.287403 23627 ts_tablet_manager.cc:1939] T 6a4a5b3a31b045619ed78ae7d24c9b1a P c6c619cf4dcb40d38bcc6bb630ec32cd: Deleting consensus metadata
I20260811 04:40:29.288420 23628 ts_tablet_manager.cc:1918] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:29.288421 23073 catalog_manager.cc:4841] TS c6c619cf4dcb40d38bcc6bb630ec32cd (127.2.238.131:44863): tablet 6a4a5b3a31b045619ed78ae7d24c9b1a (table test-table [id=9d36033fc162472285bbd5edc7f49519]) successfully deleted
I20260811 04:40:29.288503 23628 log.cc:1196] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/6a4a5b3a31b045619ed78ae7d24c9b1a
I20260811 04:40:29.288808 23628 ts_tablet_manager.cc:1939] T 6a4a5b3a31b045619ed78ae7d24c9b1a P 84b97b1792f94a88bdf55afcf387b502: Deleting consensus metadata
I20260811 04:40:29.289873 23071 catalog_manager.cc:4841] TS 84b97b1792f94a88bdf55afcf387b502 (127.2.238.130:34575): tablet 6a4a5b3a31b045619ed78ae7d24c9b1a (table test-table [id=9d36033fc162472285bbd5edc7f49519]) successfully deleted
W20260811 04:40:29.295197 23072 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:33359 (error 108)
I20260811 04:40:29.295360 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23296
I20260811 04:40:29.296950 23597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:29.296510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:33359 (local address 127.0.0.1:36104)
0811 04:40:29.296775 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:29.296851 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
W20260811 04:40:29.297106 23072 catalog_manager.cc:4584] TS 3876a97eb62f4e1f8c66b3e140142d70 (127.2.238.129:33359): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a4a5b3a31b045619ed78ae7d24c9b1a: Network error: Client connection negotiation failed: client connection to 127.2.238.129:33359: connect: Connection refused (error 111)
I20260811 04:40:29.302728 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23438
I20260811 04:40:29.310868 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23055
2026-08-11T04:40:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1251 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:29 dist-test-slave-0kls krb5kdc[23634](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:29 dist-test-slave-0kls krb5kdc[23634](info): set up 2 sockets
Aug 11 04:40:29 dist-test-slave-0kls krb5kdc[23634](info): commencing operation
krb5kdc: starting...
W20260811 04:40:30.370059 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:30 dist-test-slave-0kls krb5kdc[23634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:30.494177 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34763
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:34763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:30.606577 23651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:30.606827 23651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:30.606868 23651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:30.610781 23651 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:30.610831 23651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:30.610852 23651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:30.610869 23651 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:30.610886 23651 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:30.615862 23651 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:34763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:34763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:30.616755 23651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:30.617650 23651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:30.625710 23660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:30.625718 23658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:30.625705 23657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:30.626075 23651 server_base.cc:1034] running on GCE node
I20260811 04:40:30.626639 23651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:30.627617 23651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:30.628816 23651 hybrid_clock.cc:648] HybridClock initialized: now 1786423230628789 us; error 30 us; skew 500 ppm
Aug 11 04:40:30 dist-test-slave-0kls krb5kdc[23634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:30.631714 23651 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:30.632627 23651 webserver.cc:460] Webserver started at http://127.2.238.190:33331/ using document root <none> and password file <none>
I20260811 04:40:30.633191 23651 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:30.633260 23651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:30.633531 23651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:30.635720 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "5c9579d159e248988851de71af92a765"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "0cc2e157d16a319d703aa7d96b8274d8"
server_key_iv: "0a2e4a67a3c227af54bdd25a254352e0"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.636354 23651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "5c9579d159e248988851de71af92a765"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "0cc2e157d16a319d703aa7d96b8274d8"
server_key_iv: "0a2e4a67a3c227af54bdd25a254352e0"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.640509 23651 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:40:30.643057 23667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:30.644351 23651 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:40:30.644495 23651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "5c9579d159e248988851de71af92a765"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "0cc2e157d16a319d703aa7d96b8274d8"
server_key_iv: "0a2e4a67a3c227af54bdd25a254352e0"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.644611 23651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:30.665220 23651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:30.668085 23651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:30.668298 23651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:30.675908 23651 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:34763
I20260811 04:40:30.675933 23722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:34763 every 8 connection(s)
I20260811 04:40:30.676856 23651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:30.679623 23723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:30.680948 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23651
I20260811 04:40:30.681113 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:30.681427 3002 external_mini_cluster.cc:1428] Setting key 26e8cb7dfb401bb75a108df341a85ef2
I20260811 04:40:30.685968 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: Bootstrap starting.
I20260811 04:40:30.688246 23723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:30.689177 23723 log.cc:824] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:40:30 dist-test-slave-0kls krb5kdc[23634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:30.691262 23723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: No bootstrap required, opened a new log
I20260811 04:40:30.694841 23723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } }
I20260811 04:40:30.695086 23723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:30.695171 23723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c9579d159e248988851de71af92a765, State: Initialized, Role: FOLLOWER
I20260811 04:40:30.695654 23723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } }
I20260811 04:40:30.695796 23723 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:30.695878 23723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:30.695960 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:30.682949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.2.238.190:34763)
0811 04:40:30.683408 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:30.683423 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:30.683459 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:40:30.684208 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:30.684227 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:30.684569 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:30.684905 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:30.685775 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.686904 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.687792 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.688276 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.688455 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:30.691254 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:30.691259 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:30.691271 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:30.691297 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:30.692910 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:30.693330 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:30.693334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:30.693338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:30.693455 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:30.693815 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:30.693820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:30.693825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:30.694184 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:30.694426 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:30.694671 (+ 245us) server_negotiation.cc:299] Negotiation successful
0811 04:40:30.695017 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":162,"threads_started":1}
I20260811 04:40:30.695971 23723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:30.697045 23723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } }
I20260811 04:40:30.697405 23723 leader_election.cc:302] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c9579d159e248988851de71af92a765; no voters:
I20260811 04:40:30.697744 23723 leader_election.cc:288] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:30.697863 23730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:30.698055 23730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [term 1 LEADER]: Becoming Leader. State: Replica: 5c9579d159e248988851de71af92a765, State: Running, Role: LEADER
I20260811 04:40:30.698339 23730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } }
I20260811 04:40:30.699013 23723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:30.699506 23730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c9579d159e248988851de71af92a765. Latest consensus state: current_term: 1 leader_uuid: "5c9579d159e248988851de71af92a765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } } }
I20260811 04:40:30.699621 23730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:30.700070 23731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c9579d159e248988851de71af92a765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c9579d159e248988851de71af92a765" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 34763 } } }
I20260811 04:40:30.700183 23731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765 [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:30.703972 23746 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:30.704074 23746 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:30.704154 23734 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:30.705042 23734 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:30.710708 23734 catalog_manager.cc:1347] Generated new cluster ID: 0357665880c94e3db7e998fed35ba092
I20260811 04:40:30.710765 23734 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:30.731594 23734 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:30.732861 23734 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:30.740908 23734 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: Generated new TSK 0
I20260811 04:40:30.741684 23734 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.238.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
Entry for principal kudu/127.2.238.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab.
I20260811 04:40:30.772830 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:30.888878 23752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:30.889153 23752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:30.889230 23752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:30.893067 23752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:30.893175 23752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:30.893263 23752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:30.898263 23752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.129.keytab
--principal=kudu/127.2.238.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23752
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:30.899267 23752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:30.900408 23752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:30.910286 23758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:30.910300 23761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:30.910300 23759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:30.910394 23752 server_base.cc:1034] running on GCE node
I20260811 04:40:30.911028 23752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:30.911664 23752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:30.912868 23752 hybrid_clock.cc:648] HybridClock initialized: now 1786423230912847 us; error 36 us; skew 500 ppm
Aug 11 04:40:30 dist-test-slave-0kls krb5kdc[23634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:30.915643 23752 init.cc:377] Logged in from keytab as kudu/127.2.238.129@KRBTEST.COM (short username kudu)
I20260811 04:40:30.916456 23752 webserver.cc:460] Webserver started at http://127.2.238.129:35909/ using document root <none> and password file <none>
I20260811 04:40:30.916962 23752 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:30.917018 23752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:30.917181 23752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:30.919196 23752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "d2f398c174c049519cf560e7f4998dea"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "178ba1de64db0895222f5388db0841bf"
server_key_iv: "8f5bb1faa1ebe19d728b572e607f9960"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.919795 23752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "d2f398c174c049519cf560e7f4998dea"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "178ba1de64db0895222f5388db0841bf"
server_key_iv: "8f5bb1faa1ebe19d728b572e607f9960"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.923519 23752 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:40:30.925859 23768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:30.927053 23752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:30.927172 23752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "d2f398c174c049519cf560e7f4998dea"
format_stamp: "Formatted at 2026-08-11 04:40:30 on dist-test-slave-0kls"
server_key: "178ba1de64db0895222f5388db0841bf"
server_key_iv: "8f5bb1faa1ebe19d728b572e607f9960"
server_key_version: "encryptionkey@0"
I20260811 04:40:30.927279 23752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:30.937291 23752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:30.940124 23752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:30.940330 23752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:30.940905 23752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:30.941704 23752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:30.941749 23752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:30.941831 23752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:30.941872 23752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:30.950843 23752 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:35637
I20260811 04:40:30.950883 23897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:35637 every 8 connection(s)
I20260811 04:40:30.951676 23752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:30.960644 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23752
I20260811 04:40:30.960790 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
Aug 11 04:40:30 dist-test-slave-0kls krb5kdc[23634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.129@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:30.961149 3002 external_mini_cluster.cc:1428] Setting key 3da18bf44ef122bf080579a2f1226b95
I20260811 04:40:30.965237 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:30.953485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:51509 (local address 127.2.238.190:34763)
0811 04:40:30.953634 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:30.953638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:30.954578 (+ 940us) server_negotiation.cc:407] Connection header received
0811 04:40:30.955835 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:30.955837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:30.955884 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:30.955956 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:30.957117 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.957634 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.958797 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.959030 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.959113 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:30.961540 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:30.961543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:30.961545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:30.961577 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:30.963003 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:30.963569 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:30.963571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:30.963572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:30.963648 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:30.964006 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:30.964008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:30.964009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:30.964161 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:30.964229 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:30.964896 (+ 667us) server_negotiation.cc:299] Negotiation successful
0811 04:40:30.965037 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260811 04:40:30.966023 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:30.953927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34763 (local address 127.2.238.129:51509)
0811 04:40:30.954411 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:30.954461 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:30.955633 (+ 1172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:30.956121 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:30.956127 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:30.956455 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:30.956981 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:30.956996 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.957746 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.957748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:30.958691 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:30.958701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:30.959158 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:30.959160 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:30.959300 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:30.959812 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:30.959845 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:30.961375 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:30.963119 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:30.963127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:30.963142 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:30.963479 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:30.963732 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:30.963737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:30.963741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:30.963919 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:30.964255 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:30.964263 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:30.964610 (+ 347us) client_negotiation.cc:769] Sending connection context
0811 04:40:30.964817 (+ 207us) client_negotiation.cc:240] Negotiation successful
0811 04:40:30.965058 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":117,"threads_started":1}
I20260811 04:40:30.967255 23898 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34763
I20260811 04:40:30.967504 23898 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:30.968024 23898 heartbeater.cc:511] Master 127.2.238.190:34763 requested a full tablet report, sending...
I20260811 04:40:30.969722 23685 ts_manager.cc:194] Registered new tserver with Master: d2f398c174c049519cf560e7f4998dea (127.2.238.129:35637)
I20260811 04:40:30.971479 23685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.129@KRBTEST.COM'} at 127.2.238.129:51509
WARNING: no policy specified for kudu/127.2.238.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
Entry for principal kudu/127.2.238.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab.
I20260811 04:40:30.989928 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:31.100165 23904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:31.100423 23904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:31.100464 23904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:31.104560 23904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:31.104627 23904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:31.104719 23904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:31.109829 23904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.130.keytab
--principal=kudu/127.2.238.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:31.110750 23904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:31.111658 23904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:31.120728 23910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:31.120733 23909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:31.120837 23904 server_base.cc:1034] running on GCE node
W20260811 04:40:31.120733 23912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:31.121388 23904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:31.122042 23904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:31.123196 23904 hybrid_clock.cc:648] HybridClock initialized: now 1786423231123184 us; error 22 us; skew 500 ppm
Aug 11 04:40:31 dist-test-slave-0kls krb5kdc[23634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:31.125753 23904 init.cc:377] Logged in from keytab as kudu/127.2.238.130@KRBTEST.COM (short username kudu)
I20260811 04:40:31.126556 23904 webserver.cc:460] Webserver started at http://127.2.238.130:41855/ using document root <none> and password file <none>
I20260811 04:40:31.127085 23904 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:31.127170 23904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:31.127393 23904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:31.129318 23904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "b4bbfe35b27b4ab4891feeee04d37838"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "7abdbe0710d0c815118c3f8fadf616f0"
server_key_iv: "afd289ec7aa727adfb59e7f41cee3316"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.129957 23904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "b4bbfe35b27b4ab4891feeee04d37838"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "7abdbe0710d0c815118c3f8fadf616f0"
server_key_iv: "afd289ec7aa727adfb59e7f41cee3316"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.133265 23904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:40:31.135725 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.136847 23904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:31.136986 23904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "b4bbfe35b27b4ab4891feeee04d37838"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "7abdbe0710d0c815118c3f8fadf616f0"
server_key_iv: "afd289ec7aa727adfb59e7f41cee3316"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.137092 23904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:31.156046 23904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:31.158893 23904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:31.159098 23904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:31.159683 23904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:31.160527 23904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:31.160601 23904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.160686 23904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:31.160732 23904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.169525 23904 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:36197
I20260811 04:40:31.169582 24044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:36197 every 8 connection(s)
I20260811 04:40:31.170490 23904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:31.175981 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 23904
I20260811 04:40:31.176106 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:31.176404 3002 external_mini_cluster.cc:1428] Setting key 5097942d3afae23f3ba615a587dc3cda
Aug 11 04:40:31 dist-test-slave-0kls krb5kdc[23634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.130@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:31.185205 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.172288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:52729 (local address 127.2.238.190:34763)
0811 04:40:31.172421 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.172425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.173208 (+ 783us) server_negotiation.cc:407] Connection header received
0811 04:40:31.174549 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.174552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.174626 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.174728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:31.175912 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.176542 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.177889 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.178089 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.178171 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.181425 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:31.181428 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:31.181430 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:31.181452 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:31.183163 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.183688 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.183691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.183692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.183766 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.184112 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.184115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.184116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.184306 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:31.184435 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.184904 (+ 469us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.185035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:40:31.186156 24047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.172581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34763 (local address 127.2.238.130:52729)
0811 04:40:31.173030 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.173070 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.174334 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.174903 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.174910 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.175257 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:31.175756 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.175775 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.176663 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.176667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.177737 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.177747 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.178244 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.178248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.178486 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.179129 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:31.179153 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:31.181254 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:31.183277 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:31.183282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:31.183293 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:31.183536 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:31.183867 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:31.183870 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:31.183872 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:31.184004 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:31.184435 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:31.184442 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:31.184670 (+ 228us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.184858 (+ 188us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.185181 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260811 04:40:31.187371 24045 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34763
WARNING: no policy specified for kudu/127.2.238.131@KRBTEST.COM; defaulting to no policy
I20260811 04:40:31.187647 24045 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:31.188187 24045 heartbeater.cc:511] Master 127.2.238.190:34763 requested a full tablet report, sending...
I20260811 04:40:31.189230 23685 ts_manager.cc:194] Registered new tserver with Master: b4bbfe35b27b4ab4891feeee04d37838 (127.2.238.130:36197)
I20260811 04:40:31.189746 23685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.130@KRBTEST.COM'} at 127.2.238.130:52729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
Entry for principal kudu/127.2.238.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab.
I20260811 04:40:31.205499 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:39343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:31.316284 24050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:31.316520 24050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:31.316562 24050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:31.320770 24050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:31.320827 24050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:31.320936 24050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:31.326290 24050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:39343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.131.keytab
--principal=kudu/127.2.238.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:34763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:31.327389 24050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:31.328504 24050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:31.338281 24058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:31.338382 24050 server_base.cc:1034] running on GCE node
W20260811 04:40:31.338289 24056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:31.338274 24055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:31.338884 24050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:31.339491 24050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:31.340648 24050 hybrid_clock.cc:648] HybridClock initialized: now 1786423231340617 us; error 40 us; skew 500 ppm
Aug 11 04:40:31 dist-test-slave-0kls krb5kdc[23634](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:31.343503 24050 init.cc:377] Logged in from keytab as kudu/127.2.238.131@KRBTEST.COM (short username kudu)
I20260811 04:40:31.344380 24050 webserver.cc:460] Webserver started at http://127.2.238.131:36895/ using document root <none> and password file <none>
I20260811 04:40:31.344933 24050 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:31.344991 24050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:31.345229 24050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:31.347311 24050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "683b90b34cce4e1ba9bee37c0041fe63"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "ebd25c83b9830453fbf73160c3971138"
server_key_iv: "9af83a38288b758b79f90ad8e49ceb07"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.347895 24050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "683b90b34cce4e1ba9bee37c0041fe63"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "ebd25c83b9830453fbf73160c3971138"
server_key_iv: "9af83a38288b758b79f90ad8e49ceb07"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.351523 24050 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260811 04:40:31.353940 24066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.355121 24050 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:31.355255 24050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "683b90b34cce4e1ba9bee37c0041fe63"
format_stamp: "Formatted at 2026-08-11 04:40:31 on dist-test-slave-0kls"
server_key: "ebd25c83b9830453fbf73160c3971138"
server_key_iv: "9af83a38288b758b79f90ad8e49ceb07"
server_key_version: "encryptionkey@0"
I20260811 04:40:31.355362 24050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:31.372774 24050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:31.375679 24050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:31.375880 24050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:31.376475 24050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:31.377264 24050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:31.377336 24050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.377403 24050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:31.377434 24050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:31.386314 24050 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:45657
I20260811 04:40:31.386338 24193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:45657 every 8 connection(s)
I20260811 04:40:31.387144 24050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:31.392578 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24050
I20260811 04:40:31.392714 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:31.392970 3002 external_mini_cluster.cc:1428] Setting key c1f876a993a92e79d1dd1b4ae9bd3b12
Aug 11 04:40:31 dist-test-slave-0kls krb5kdc[23634](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.131@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:31.400686 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.388938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:33371 (local address 127.2.238.190:34763)
0811 04:40:31.389092 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.389096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.389931 (+ 835us) server_negotiation.cc:407] Connection header received
0811 04:40:31.391146 (+ 1215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.391149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.391204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.391307 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:31.392362 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.392920 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.394234 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.394468 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.394580 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.396930 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:31.396933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:31.396936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:31.396965 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:31.398465 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.399031 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.399035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.399036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.399118 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.399456 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.399460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.399461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.399692 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:31.399809 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.400385 (+ 576us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.400504 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 04:40:31.401530 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.389281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:34763 (local address 127.2.238.131:33371)
0811 04:40:31.389756 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.389797 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.390930 (+ 1133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.391467 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.391473 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.391786 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:31.392221 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.392234 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.393032 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.393037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.394116 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.394126 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.394619 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.394623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.394795 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.395388 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:40:31.395405 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:31.396782 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:31.398591 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:31.398596 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:31.398610 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:31.398932 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:31.399219 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:40:31.399222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:40:31.399224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:40:31.399362 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:40:31.399807 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:31.399813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:31.400128 (+ 315us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.400330 (+ 202us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.400646 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":110,"threads_started":1}
I20260811 04:40:31.402555 24194 heartbeater.cc:347] Connected to a master server at 127.2.238.190:34763
I20260811 04:40:31.402767 24194 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:31.403213 24194 heartbeater.cc:511] Master 127.2.238.190:34763 requested a full tablet report, sending...
I20260811 04:40:31.404140 23685 ts_manager.cc:194] Registered new tserver with Master: 683b90b34cce4e1ba9bee37c0041fe63 (127.2.238.131:45657)
I20260811 04:40:31.404699 23685 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.238.131@KRBTEST.COM'} at 127.2.238.131:33371
I20260811 04:40:31.406881 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:31.415629 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.408851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53412 (local address 127.2.238.190:34763)
0811 04:40:31.408968 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.408971 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.409196 (+ 225us) server_negotiation.cc:407] Connection header received
0811 04:40:31.409281 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.409283 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.409320 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.409383 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:31.410139 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.410599 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.411352 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.411553 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.411621 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.412871 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:31.412874 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:31.412876 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:31.412897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:31.414292 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.414661 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.414664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.414665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.414735 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:31.414958 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:31.414960 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:31.414961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:31.415165 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:31.415266 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.415424 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.415509 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:40:31.418426 23685 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:31.421602 23685 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:31.436375 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:36197 (local address 127.0.0.1:33314)
0811 04:40:31.431625 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.431635 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.431701 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.432159 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.432165 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.432198 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:31.432368 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.432376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433902 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.433908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.434812 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.434822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.435797 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.435804 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.435923 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.435960 (+ 37us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.436075 (+ 115us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.436208 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1166,"mutex_wait_us":24,"spinlock_wait_cycles":2304,"thread_start_us":105,"threads_started":1}
I20260811 04:40:31.436393 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33314 (local address 127.2.238.130:36197)
0811 04:40:31.431290 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.431304 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.431713 (+ 409us) server_negotiation.cc:407] Connection header received
0811 04:40:31.431809 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.431814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.432012 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.432150 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:31.432516 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433698 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.434940 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.435664 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.435810 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.435975 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.436121 (+ 146us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.436217 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":99,"threads_started":1}
I20260811 04:40:31.436722 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52512 (local address 127.2.238.131:45657)
0811 04:40:31.431512 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.431520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.431539 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:40:31.431598 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.431601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.431811 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.431981 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:31.432514 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433312 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.435025 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.435713 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.435835 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.435987 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.436442 (+ 455us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.436562 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":122,"threads_started":1}
I20260811 04:40:31.436378 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45657 (local address 127.0.0.1:52512)
0811 04:40:31.431337 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.431349 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.431427 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.432169 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.432173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.432204 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:31.432359 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.432369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433438 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.433463 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.434881 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.434892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.436051 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.436056 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.436140 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.436160 (+ 20us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.436224 (+ 64us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.436287 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1153,"mutex_wait_us":3,"spinlock_wait_cycles":2816,"thread_start_us":80,"threads_started":1}
I20260811 04:40:31.437139 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.2.238.129:35637)
0811 04:40:31.431991 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.432003 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.432020 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:40:31.432077 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.432080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.432261 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.432426 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:31.432645 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433763 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.435402 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.436236 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.436344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.436476 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.436921 (+ 445us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.437005 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1746,"thread_start_us":102,"threads_started":1}
I20260811 04:40:31.437358 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.430043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:35637 (local address 127.0.0.1:42060)
0811 04:40:31.430644 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.430697 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.430829 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.432396 (+ 1567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.432401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.432441 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:31.432526 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.432531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.433936 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.433939 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.435273 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.435283 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.436539 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.436542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.436626 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.436647 (+ 21us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.436717 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.436786 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":224,"thread_start_us":83,"threads_started":1}
I20260811 04:40:31.439368 23971 tablet_service.cc:1467] Processing CreateTablet for tablet 786b54044a6240ec85e0adefd73faf87 (DEFAULT_TABLE table=test-table [id=8fa5237f9d694b7a955cf9bec94dcd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:31.439597 24118 tablet_service.cc:1467] Processing CreateTablet for tablet 786b54044a6240ec85e0adefd73faf87 (DEFAULT_TABLE table=test-table [id=8fa5237f9d694b7a955cf9bec94dcd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:31.439742 23823 tablet_service.cc:1467] Processing CreateTablet for tablet 786b54044a6240ec85e0adefd73faf87 (DEFAULT_TABLE table=test-table [id=8fa5237f9d694b7a955cf9bec94dcd1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:31.440532 23971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b54044a6240ec85e0adefd73faf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:31.440724 24118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b54044a6240ec85e0adefd73faf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:31.440721 23823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 786b54044a6240ec85e0adefd73faf87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:31.446940 24215 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Bootstrap starting.
I20260811 04:40:31.448516 24217 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Bootstrap starting.
I20260811 04:40:31.449779 24215 tablet_bootstrap.cc:654] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:31.450124 24216 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Bootstrap starting.
I20260811 04:40:31.450297 24217 tablet_bootstrap.cc:654] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:31.450623 24215 log.cc:824] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:31.451553 24217 log.cc:824] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:31.452299 24216 tablet_bootstrap.cc:654] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:31.452864 24215 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: No bootstrap required, opened a new log
I20260811 04:40:31.453068 24215 ts_tablet_manager.cc:1397] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260811 04:40:31.453473 24216 log.cc:824] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:31.453752 24217 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: No bootstrap required, opened a new log
I20260811 04:40:31.453972 24217 ts_tablet_manager.cc:1397] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260811 04:40:31.456101 24215 raft_consensus.cc:348] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.456363 24215 raft_consensus.cc:374] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:31.456440 24215 raft_consensus.cc:729] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4bbfe35b27b4ab4891feeee04d37838, State: Initialized, Role: FOLLOWER
I20260811 04:40:31.456938 24215 consensus_queue.cc:260] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.456840 24217 raft_consensus.cc:348] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.457129 24217 raft_consensus.cc:374] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:31.457177 24217 raft_consensus.cc:729] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 683b90b34cce4e1ba9bee37c0041fe63, State: Initialized, Role: FOLLOWER
I20260811 04:40:31.457573 24217 consensus_queue.cc:260] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.457697 24045 heartbeater.cc:503] Master 127.2.238.190:34763 was elected leader, sending a full tablet report...
I20260811 04:40:31.458189 24216 tablet_bootstrap.cc:492] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: No bootstrap required, opened a new log
I20260811 04:40:31.458240 24194 heartbeater.cc:503] Master 127.2.238.190:34763 was elected leader, sending a full tablet report...
I20260811 04:40:31.458374 24216 ts_tablet_manager.cc:1397] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20260811 04:40:31.459283 24217 ts_tablet_manager.cc:1428] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260811 04:40:31.460862 24215 ts_tablet_manager.cc:1428] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260811 04:40:31.462236 24216 raft_consensus.cc:348] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.462513 24216 raft_consensus.cc:374] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:31.462584 24216 raft_consensus.cc:729] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2f398c174c049519cf560e7f4998dea, State: Initialized, Role: FOLLOWER
I20260811 04:40:31.463034 24216 consensus_queue.cc:260] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.463706 23898 heartbeater.cc:503] Master 127.2.238.190:34763 was elected leader, sending a full tablet report...
I20260811 04:40:31.463948 24216 ts_tablet_manager.cc:1428] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260811 04:40:31.638047 24196 tablet.cc:2369] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:31.671710 24046 tablet.cc:2369] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:31.703035 23899 tablet.cc:2369] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:40:31.824213 24223 raft_consensus.cc:482] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:31.824487 24223 raft_consensus.cc:504] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.825675 24223 leader_election.cc:288] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 683b90b34cce4e1ba9bee37c0041fe63 (127.2.238.131:45657), b4bbfe35b27b4ab4891feeee04d37838 (127.2.238.130:36197)
I20260811 04:40:31.829891 24226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.826005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:36197 (local address 127.2.238.129:40181)
0811 04:40:31.826439 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.826451 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.826570 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.826887 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.826891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.826912 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:31.827061 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.827067 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.827815 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.827818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.828710 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.828720 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.829411 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.829415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.829512 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.829535 (+ 23us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.829604 (+ 69us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.829669 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":44,"thread_start_us":109,"threads_started":1}
I20260811 04:40:31.829946 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.826239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:40181 (local address 127.2.238.130:36197)
0811 04:40:31.826379 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.826384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.826566 (+ 182us) server_negotiation.cc:407] Connection header received
0811 04:40:31.826707 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.826712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.826773 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.826845 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:31.827169 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.827670 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.828844 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.829273 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.829402 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.829468 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.829647 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.829735 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260811 04:40:31.829984 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.826132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:45051 (local address 127.2.238.131:45657)
0811 04:40:31.826283 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.826287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.826635 (+ 348us) server_negotiation.cc:407] Connection header received
0811 04:40:31.826779 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.826782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.826847 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.826922 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:31.827236 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.827671 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.828908 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.829300 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.829437 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.829491 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.829741 (+ 250us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.829850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260811 04:40:31.830056 24224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.826005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45657 (local address 127.2.238.129:45051)
0811 04:40:31.826489 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.826535 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:31.826671 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:31.826963 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:31.826966 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:31.826990 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:31.827098 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.827103 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.827800 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.827805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.828748 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:31.828759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.829459 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.829463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:31.829576 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.829625 (+ 49us) client_negotiation.cc:769] Sending connection context
0811 04:40:31.829713 (+ 88us) client_negotiation.cc:240] Negotiation successful
0811 04:40:31.829806 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":117,"thread_start_us":140,"threads_started":1}
I20260811 04:40:31.830580 23993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b54044a6240ec85e0adefd73faf87" candidate_uuid: "d2f398c174c049519cf560e7f4998dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bbfe35b27b4ab4891feeee04d37838" is_pre_election: true
I20260811 04:40:31.830626 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b54044a6240ec85e0adefd73faf87" candidate_uuid: "d2f398c174c049519cf560e7f4998dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683b90b34cce4e1ba9bee37c0041fe63" is_pre_election: true
I20260811 04:40:31.830914 24142 raft_consensus.cc:2454] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f398c174c049519cf560e7f4998dea in term 0.
I20260811 04:40:31.830924 23993 raft_consensus.cc:2454] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2f398c174c049519cf560e7f4998dea in term 0.
I20260811 04:40:31.831396 23772 leader_election.cc:302] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b4bbfe35b27b4ab4891feeee04d37838, d2f398c174c049519cf560e7f4998dea; no voters:
I20260811 04:40:31.831694 24223 raft_consensus.cc:2781] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:31.831776 24223 raft_consensus.cc:482] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:31.831817 24223 raft_consensus.cc:3037] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:31.832862 24223 raft_consensus.cc:504] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.833190 24223 leader_election.cc:288] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [CANDIDATE]: Term 1 election: Requested vote from peers 683b90b34cce4e1ba9bee37c0041fe63 (127.2.238.131:45657), b4bbfe35b27b4ab4891feeee04d37838 (127.2.238.130:36197)
I20260811 04:40:31.833570 24142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b54044a6240ec85e0adefd73faf87" candidate_uuid: "d2f398c174c049519cf560e7f4998dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "683b90b34cce4e1ba9bee37c0041fe63"
I20260811 04:40:31.833592 23993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "786b54044a6240ec85e0adefd73faf87" candidate_uuid: "d2f398c174c049519cf560e7f4998dea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4bbfe35b27b4ab4891feeee04d37838"
I20260811 04:40:31.833671 24142 raft_consensus.cc:3037] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:31.833724 23993 raft_consensus.cc:3037] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:31.834712 24142 raft_consensus.cc:2454] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f398c174c049519cf560e7f4998dea in term 1.
I20260811 04:40:31.834913 23993 raft_consensus.cc:2454] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2f398c174c049519cf560e7f4998dea in term 1.
I20260811 04:40:31.835093 23769 leader_election.cc:302] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 683b90b34cce4e1ba9bee37c0041fe63, d2f398c174c049519cf560e7f4998dea; no voters:
I20260811 04:40:31.835306 24223 raft_consensus.cc:2781] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:31.835573 24223 raft_consensus.cc:686] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 1 LEADER]: Becoming Leader. State: Replica: d2f398c174c049519cf560e7f4998dea, State: Running, Role: LEADER
I20260811 04:40:31.835906 24223 consensus_queue.cc:237] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } }
I20260811 04:40:31.839424 23685 catalog_manager.cc:5495] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea reported cstate change: term changed from 0 to 1, leader changed from <none> to d2f398c174c049519cf560e7f4998dea (127.2.238.129). New cstate: current_term: 1 leader_uuid: "d2f398c174c049519cf560e7f4998dea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2f398c174c049519cf560e7f4998dea" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 35637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 } health_report { overall_health: UNKNOWN } } }
I20260811 04:40:31.886941 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.883539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.2.238.129:35637)
0811 04:40:31.883717 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:31.883722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:31.883741 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:40:31.883864 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:31.883868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:31.883960 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:31.884091 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:31.884324 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.884826 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.885625 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:31.885846 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:31.885920 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:31.886208 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:31.886582 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:31.886669 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:31.886730 (+ 61us) server_negotiation.cc:299] Negotiation successful
0811 04:40:31.886775 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260811 04:40:31.892807 23993 raft_consensus.cc:1261] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 1 FOLLOWER]: Refusing update from remote peer d2f398c174c049519cf560e7f4998dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:31.892808 24142 raft_consensus.cc:1261] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 1 FOLLOWER]: Refusing update from remote peer d2f398c174c049519cf560e7f4998dea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:31.893544 24223 consensus_queue.cc:1035] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "683b90b34cce4e1ba9bee37c0041fe63" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:31.893813 24227 consensus_queue.cc:1035] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4bbfe35b27b4ab4891feeee04d37838" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 36197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:31.899982 24230 mvcc.cc:205] Tried to move back new op lower bound from 7317189557825413120 to 7317189557602201600. Current Snapshot: MvccSnapshot[applied={T|T < 7317189557825413120}]
I20260811 04:40:31.902575 24231 mvcc.cc:205] Tried to move back new op lower bound from 7317189557825413120 to 7317189557602201600. Current Snapshot: MvccSnapshot[applied={T|T < 7317189557825413120}]
I20260811 04:40:31.910506 23684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:31.910667 23684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:31.912853 23684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c9579d159e248988851de71af92a765: Sending DeleteTablet for 3 replicas of tablet 786b54044a6240ec85e0adefd73faf87
I20260811 04:40:31.913528 24118 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b54044a6240ec85e0adefd73faf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:31 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:52512
I20260811 04:40:31.913545 23823 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b54044a6240ec85e0adefd73faf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:31 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:42060
I20260811 04:40:31.913727 23971 tablet_service.cc:1514] Processing DeleteTablet for tablet 786b54044a6240ec85e0adefd73faf87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:31 UTC) from {username='kudu', principal='kudu/127.2.238.190@KRBTEST.COM'} at 127.0.0.1:33314
I20260811 04:40:31.914204 24242 tablet_replica.cc:330] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: stopping tablet replica
I20260811 04:40:31.914258 24241 tablet_replica.cc:330] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea: stopping tablet replica
I20260811 04:40:31.914526 24242 raft_consensus.cc:2229] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:31.914551 24241 raft_consensus.cc:2229] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:31.914835 24242 raft_consensus.cc:2258] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:31.914923 24241 raft_consensus.cc:2258] T 786b54044a6240ec85e0adefd73faf87 P d2f398c174c049519cf560e7f4998dea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:31.914934 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23752
I20260811 04:40:31.916656 24240 tablet_replica.cc:330] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: stopping tablet replica
I20260811 04:40:31.917039 24240 raft_consensus.cc:2229] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:31.917393 24242 ts_tablet_manager.cc:1905] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:31.917407 24240 raft_consensus.cc:2258] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:31.919477 24240 ts_tablet_manager.cc:1905] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:31.921150 24242 ts_tablet_manager.cc:1918] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:31.921272 24242 log.cc:1196] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-1/wal/wals/786b54044a6240ec85e0adefd73faf87
I20260811 04:40:31.921566 24242 ts_tablet_manager.cc:1939] T 786b54044a6240ec85e0adefd73faf87 P b4bbfe35b27b4ab4891feeee04d37838: Deleting consensus metadata
I20260811 04:40:31.922818 23671 catalog_manager.cc:4841] TS b4bbfe35b27b4ab4891feeee04d37838 (127.2.238.130:36197): tablet 786b54044a6240ec85e0adefd73faf87 (table test-table [id=8fa5237f9d694b7a955cf9bec94dcd1f]) successfully deleted
I20260811 04:40:31.923231 24240 ts_tablet_manager.cc:1918] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:31.923339 24240 log.cc:1196] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/786b54044a6240ec85e0adefd73faf87
I20260811 04:40:31.923666 24240 ts_tablet_manager.cc:1939] T 786b54044a6240ec85e0adefd73faf87 P 683b90b34cce4e1ba9bee37c0041fe63: Deleting consensus metadata
I20260811 04:40:31.925105 23668 catalog_manager.cc:4841] TS 683b90b34cce4e1ba9bee37c0041fe63 (127.2.238.131:45657): tablet 786b54044a6240ec85e0adefd73faf87 (table test-table [id=8fa5237f9d694b7a955cf9bec94dcd1f]) successfully deleted
W20260811 04:40:31.928210 23670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:35637 (error 108)
I20260811 04:40:31.929064 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23904
I20260811 04:40:31.929253 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:31.928911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:35637 (local address 127.0.0.1:42090)
0811 04:40:31.929050 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:31.929150 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:35637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:40:31.929432 23670 catalog_manager.cc:4584] TS d2f398c174c049519cf560e7f4998dea (127.2.238.129:35637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 786b54044a6240ec85e0adefd73faf87: Network error: Client connection negotiation failed: client connection to 127.2.238.129:35637: connect: Connection refused (error 111)
I20260811 04:40:31.937176 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24050
I20260811 04:40:31.943609 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 23651
2026-08-11T04:40:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2632 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-11T04:40:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:31Z Disabled control of system clock
I20260811 04:40:31.975808 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42849
--webserver_interface=127.2.238.190
--webserver_port=0
--builtin_ntp_servers=127.2.238.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:42849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:32.089275 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:32.089501 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:32.089533 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:32.093364 24248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:32.093416 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:32.093430 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:32.093444 24248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:32.093458 24248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:32.099164 24248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:42849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:32.100245 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:32.101637 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:32.110083 24257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.110092 24255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.110102 24254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:32.110358 24248 server_base.cc:1034] running on GCE node
I20260811 04:40:32.111052 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:32.112082 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:32.113281 24248 hybrid_clock.cc:648] HybridClock initialized: now 1786423232113273 us; error 28 us; skew 500 ppm
I20260811 04:40:32.115147 24248 webserver.cc:460] Webserver started at http://127.2.238.190:34271/ using document root <none> and password file <none>
I20260811 04:40:32.115724 24248 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:32.115813 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:32.116055 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:32.118110 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "fb25bbf3b3c84f6b9d0d8ca287003586"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "5c9b862dfb73fccedb35c43417f3ffb6"
server_key_iv: "d1abe6131f6f8ea0a07e4c6a7dac4993"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.118674 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "fb25bbf3b3c84f6b9d0d8ca287003586"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "5c9b862dfb73fccedb35c43417f3ffb6"
server_key_iv: "d1abe6131f6f8ea0a07e4c6a7dac4993"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.122254 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260811 04:40:32.124557 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.125682 24248 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260811 04:40:32.125841 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "fb25bbf3b3c84f6b9d0d8ca287003586"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "5c9b862dfb73fccedb35c43417f3ffb6"
server_key_iv: "d1abe6131f6f8ea0a07e4c6a7dac4993"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.125952 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:32.139536 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:32.140169 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:32.140357 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:32.147363 24248 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:42849
I20260811 04:40:32.147373 24327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:42849 every 8 connection(s)
I20260811 04:40:32.148298 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:32.151325 24328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:32.151518 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24248
I20260811 04:40:32.151656 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:32.151947 3002 external_mini_cluster.cc:1428] Setting key 76b1ac07d159d6e4f11fee1e3dd9d59c
I20260811 04:40:32.158399 24328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: Bootstrap starting.
I20260811 04:40:32.160889 24328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:32.161124 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.153261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58128 (local address 127.2.238.190:42849)
0811 04:40:32.153967 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.153983 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.154017 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:40:32.154717 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.154733 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.155147 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.155417 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:32.155759 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.156691 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.157942 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.158399 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.158570 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.159378 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:32.159383 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:32.159396 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:32.159771 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:32.159837 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:32.159858 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:32.159933 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.160097 (+ 164us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.160390 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":159,"threads_started":1}
I20260811 04:40:32.161705 24328 log.cc:824] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:32.163628 24328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: No bootstrap required, opened a new log
I20260811 04:40:32.167284 24328 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } }
I20260811 04:40:32.167493 24328 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:32.167563 24328 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb25bbf3b3c84f6b9d0d8ca287003586, State: Initialized, Role: FOLLOWER
I20260811 04:40:32.168008 24328 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } }
I20260811 04:40:32.168156 24328 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:32.168242 24328 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:32.168355 24328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:32.169198 24328 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } }
I20260811 04:40:32.169530 24328 leader_election.cc:302] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb25bbf3b3c84f6b9d0d8ca287003586; no voters:
I20260811 04:40:32.169890 24328 leader_election.cc:288] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:32.170022 24334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:32.170301 24334 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [term 1 LEADER]: Becoming Leader. State: Replica: fb25bbf3b3c84f6b9d0d8ca287003586, State: Running, Role: LEADER
I20260811 04:40:32.170619 24334 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } }
I20260811 04:40:32.171193 24328 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:32.172399 24335 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } } }
I20260811 04:40:32.172448 24336 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb25bbf3b3c84f6b9d0d8ca287003586. Latest consensus state: current_term: 1 leader_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb25bbf3b3c84f6b9d0d8ca287003586" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42849 } } }
I20260811 04:40:32.172628 24336 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:32.172662 24335 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:32.173528 24345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:32.176644 24345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:32.182092 24345 catalog_manager.cc:1347] Generated new cluster ID: 4cf642fa946947bebbca9b8f2a3dae7b
I20260811 04:40:32.182163 24345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:32.194877 24345 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:32.196107 24345 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:32.207864 24345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: Generated new TSK 0
I20260811 04:40:32.208684 24345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:32.217406 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.129:0
--local_ip_for_outbound_sockets=127.2.238.129
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--builtin_ntp_servers=127.2.238.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260811 04:40:32.330974 24354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:32.331225 24354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:32.331287 24354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:32.335207 24354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:32.335279 24354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:32.335368 24354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.129
I20260811 04:40:32.340085 24354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.238.129
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.129
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:32.341142 24354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:32.342496 24354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:32.352088 24360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.352063 24359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:32.352254 24354 server_base.cc:1034] running on GCE node
W20260811 04:40:32.352322 24362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:32.352793 24354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:32.353428 24354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:32.354604 24354 hybrid_clock.cc:648] HybridClock initialized: now 1786423232354574 us; error 42 us; skew 500 ppm
I20260811 04:40:32.356334 24354 webserver.cc:460] Webserver started at http://127.2.238.129:33847/ using document root <none> and password file <none>
I20260811 04:40:32.356902 24354 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:32.356961 24354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:32.357206 24354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:32.359287 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data/instance:
uuid: "835cbd1efa444d498e431ab41c70d798"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "7a71034dc9f28763f723f33c3a1fb888"
server_key_iv: "5b1c9ed341ddc707f6098774b710fe6b"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.359917 24354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance:
uuid: "835cbd1efa444d498e431ab41c70d798"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "7a71034dc9f28763f723f33c3a1fb888"
server_key_iv: "5b1c9ed341ddc707f6098774b710fe6b"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.363477 24354 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:40:32.365706 24369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.366909 24354 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:32.367034 24354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal
uuid: "835cbd1efa444d498e431ab41c70d798"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "7a71034dc9f28763f723f33c3a1fb888"
server_key_iv: "5b1c9ed341ddc707f6098774b710fe6b"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.367134 24354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:32.390793 24354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:32.391491 24354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:32.391724 24354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:32.392379 24354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:32.393275 24354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:32.393321 24354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.393359 24354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:32.393374 24354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.402009 24354 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.129:41513
I20260811 04:40:32.402124 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.129:41513 every 8 connection(s)
I20260811 04:40:32.402930 24354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/data/info.pb
I20260811 04:40:32.404664 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24354
I20260811 04:40:32.404794 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-0/wal/instance
I20260811 04:40:32.405131 3002 external_mini_cluster.cc:1428] Setting key 505b2967e3d8ad49dd09d916103592a2
I20260811 04:40:32.407212 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.130:0
--local_ip_for_outbound_sockets=127.2.238.130
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--builtin_ntp_servers=127.2.238.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:32.413080 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.404817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.129:36619 (local address 127.2.238.190:42849)
0811 04:40:32.404981 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.404985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.406286 (+ 1301us) server_negotiation.cc:407] Connection header received
0811 04:40:32.407865 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.407871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.407933 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.408024 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:32.408910 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.409432 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.411067 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.411278 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.411369 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.412631 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:32.412634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:32.412636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:32.412648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:32.412666 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:32.412673 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:32.412726 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.412923 (+ 197us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.412975 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:40:32.414250 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.405322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42849 (local address 127.2.238.129:36619)
0811 04:40:32.406141 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.406189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.407658 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.408183 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.408189 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.408268 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:32.408730 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.408762 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.409557 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.409563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.410955 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.410968 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.411437 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.411443 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.411718 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.412334 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:32.412337 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:32.412391 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:32.412474 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:32.412766 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:32.412778 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:32.412786 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.412912 (+ 126us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.413330 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":157,"threads_started":1}
I20260811 04:40:32.415410 24502 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42849
I20260811 04:40:32.415665 24502 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:32.416128 24502 heartbeater.cc:511] Master 127.2.238.190:42849 requested a full tablet report, sending...
I20260811 04:40:32.417562 24284 ts_manager.cc:194] Registered new tserver with Master: 835cbd1efa444d498e431ab41c70d798 (127.2.238.129:41513)
I20260811 04:40:32.418735 24284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.129:36619
W20260811 04:40:32.519775 24505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:32.520076 24505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:32.520151 24505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:32.524097 24505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:32.524165 24505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:32.524247 24505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.130
I20260811 04:40:32.529008 24505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.238.130
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.130
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:32.530172 24505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:32.531518 24505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:32.541097 24513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.541097 24510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.541164 24511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:32.541213 24505 server_base.cc:1034] running on GCE node
I20260811 04:40:32.541937 24505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:32.542543 24505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:32.543737 24505 hybrid_clock.cc:648] HybridClock initialized: now 1786423232543708 us; error 46 us; skew 500 ppm
I20260811 04:40:32.545956 24505 webserver.cc:460] Webserver started at http://127.2.238.130:44091/ using document root <none> and password file <none>
I20260811 04:40:32.546578 24505 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:32.546694 24505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:32.546916 24505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:32.549045 24505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data/instance:
uuid: "a09a6f26c2554201afbd69d9f3f3f9b5"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "1a40176ed05845a9cf401c42d34267e7"
server_key_iv: "a3ff41d0986fee60e87c7b08d03a874d"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.549685 24505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance:
uuid: "a09a6f26c2554201afbd69d9f3f3f9b5"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "1a40176ed05845a9cf401c42d34267e7"
server_key_iv: "a3ff41d0986fee60e87c7b08d03a874d"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.553437 24505 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:40:32.556051 24520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.557271 24505 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:32.557418 24505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal
uuid: "a09a6f26c2554201afbd69d9f3f3f9b5"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "1a40176ed05845a9cf401c42d34267e7"
server_key_iv: "a3ff41d0986fee60e87c7b08d03a874d"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.557534 24505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:32.575315 24505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:32.575994 24505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:32.576233 24505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:32.576864 24505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:32.577927 24505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:32.578001 24505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.578094 24505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:32.578135 24505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.587731 24505 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.130:43259
I20260811 04:40:32.587757 24644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.130:43259 every 8 connection(s)
I20260811 04:40:32.588593 24505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/data/info.pb
I20260811 04:40:32.595055 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24505
I20260811 04:40:32.595211 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-1/wal/instance
I20260811 04:40:32.595614 3002 external_mini_cluster.cc:1428] Setting key 306a3d44fa726f83e56a3668f9684dcd
I20260811 04:40:32.598115 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.238.131:0
--local_ip_for_outbound_sockets=127.2.238.131
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--builtin_ntp_servers=127.2.238.148:43719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260811 04:40:32.598609 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.590580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:60047 (local address 127.2.238.190:42849)
0811 04:40:32.590734 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.590737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.591746 (+ 1009us) server_negotiation.cc:407] Connection header received
0811 04:40:32.593137 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.593139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.593201 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.593275 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:32.594273 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.594784 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.596414 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.596692 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.596771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.598052 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:32.598055 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:32.598057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:32.598069 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:32.598089 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:32.598096 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:32.598166 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.598449 (+ 283us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.598507 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:40:32.599751 24647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.590943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42849 (local address 127.2.238.130:60047)
0811 04:40:32.591581 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.591622 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.592936 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.593449 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.593455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.593549 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:32.594120 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.594135 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.594928 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.594933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.596291 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.596303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.596851 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.596856 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.597076 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.597688 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:32.597690 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:32.597757 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:32.597892 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:32.598192 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:32.598205 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:32.598213 (+ 8us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.598413 (+ 200us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.598701 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260811 04:40:32.601168 24645 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42849
I20260811 04:40:32.601406 24645 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:32.602012 24645 heartbeater.cc:511] Master 127.2.238.190:42849 requested a full tablet report, sending...
I20260811 04:40:32.603076 24284 ts_manager.cc:194] Registered new tserver with Master: a09a6f26c2554201afbd69d9f3f3f9b5 (127.2.238.130:43259)
I20260811 04:40:32.603641 24284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.130:60047
W20260811 04:40:32.714538 24648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:32.714784 24648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:32.714854 24648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:32.718884 24648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:32.718948 24648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:32.719048 24648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.238.131
I20260811 04:40:32.724167 24648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:43719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.238.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.238.131
--webserver_port=0
--tserver_master_addrs=127.2.238.190:42849
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.238.131
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:32.725332 24648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:32.726804 24648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:32.736147 24654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.736176 24655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:32.736336 24657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:32.736459 24648 server_base.cc:1034] running on GCE node
I20260811 04:40:32.736941 24648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:32.737560 24648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:32.738794 24648 hybrid_clock.cc:648] HybridClock initialized: now 1786423232738765 us; error 52 us; skew 500 ppm
I20260811 04:40:32.740566 24648 webserver.cc:460] Webserver started at http://127.2.238.131:37511/ using document root <none> and password file <none>
I20260811 04:40:32.741150 24648 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:32.741241 24648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:32.741467 24648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:32.743533 24648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data/instance:
uuid: "e3062a1ffc844e42a11db1028f5e13ab"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "e206c17d5925062aeb16aaa7e7a79c6b"
server_key_iv: "1dd54b342a3ea05030036de9073e3967"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.744155 24648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance:
uuid: "e3062a1ffc844e42a11db1028f5e13ab"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "e206c17d5925062aeb16aaa7e7a79c6b"
server_key_iv: "1dd54b342a3ea05030036de9073e3967"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.747922 24648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260811 04:40:32.750331 24663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.751550 24648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260811 04:40:32.751641 24648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal
uuid: "e3062a1ffc844e42a11db1028f5e13ab"
format_stamp: "Formatted at 2026-08-11 04:40:32 on dist-test-slave-0kls"
server_key: "e206c17d5925062aeb16aaa7e7a79c6b"
server_key_iv: "1dd54b342a3ea05030036de9073e3967"
server_key_version: "encryptionkey@0"
I20260811 04:40:32.751724 24648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:32.773141 24648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:32.773759 24648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:32.773974 24648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:32.774521 24648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260811 04:40:32.775400 24648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260811 04:40:32.775446 24648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.775484 24648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260811 04:40:32.775499 24648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:32.784471 24648 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.131:45177
I20260811 04:40:32.784499 24786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.131:45177 every 8 connection(s)
I20260811 04:40:32.785355 24648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/data/info.pb
I20260811 04:40:32.786957 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24648
I20260811 04:40:32.787070 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal/instance
I20260811 04:40:32.787386 3002 external_mini_cluster.cc:1428] Setting key c82ceb57730f2c00c13c808dcd8db641
I20260811 04:40:32.794485 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.787181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.131:49871 (local address 127.2.238.190:42849)
0811 04:40:32.787355 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.787360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.788583 (+ 1223us) server_negotiation.cc:407] Connection header received
0811 04:40:32.790210 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.790212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.790258 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.790325 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:32.791162 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.791670 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.792739 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.792965 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.793039 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.794077 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:32.794080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:32.794082 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:32.794093 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:32.794114 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:32.794122 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:32.794179 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.794312 (+ 133us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.794359 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260811 04:40:32.795342 24789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.787640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:42849 (local address 127.2.238.131:49871)
0811 04:40:32.788438 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.788493 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.790024 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.790483 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.790489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.790569 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:40:32.791043 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.791056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.791769 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.791772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.792606 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.792614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.793087 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.793090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.793229 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.793763 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0811 04:40:32.793764 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:40:32.793854 (+ 90us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0811 04:40:32.793933 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:40:32.794194 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:40:32.794201 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:40:32.794205 (+ 4us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.794296 (+ 91us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.794525 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":143,"threads_started":1}
I20260811 04:40:32.796379 24787 heartbeater.cc:347] Connected to a master server at 127.2.238.190:42849
I20260811 04:40:32.796598 24787 heartbeater.cc:464] Registering TS with master...
I20260811 04:40:32.797135 24787 heartbeater.cc:511] Master 127.2.238.190:42849 requested a full tablet report, sending...
I20260811 04:40:32.798171 24284 ts_manager.cc:194] Registered new tserver with Master: e3062a1ffc844e42a11db1028f5e13ab (127.2.238.131:45177)
I20260811 04:40:32.798727 24284 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.238.131:49871
I20260811 04:40:32.801381 3002 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260811 04:40:32.807026 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.803296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58134 (local address 127.2.238.190:42849)
0811 04:40:32.803412 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.803416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.803621 (+ 205us) server_negotiation.cc:407] Connection header received
0811 04:40:32.803770 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.803773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.803815 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.803899 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:32.804193 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.804698 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.805513 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.805750 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.805903 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.806628 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:32.806631 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:32.806634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:32.806645 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:32.806669 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:32.806677 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:32.806746 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.806857 (+ 111us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.806905 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:40:32.809432 24284 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260811 04:40:32.812489 24284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260811 04:40:32.826869 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.821054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:41513 (local address 127.0.0.1:50426)
0811 04:40:32.821587 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.821602 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.821703 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.822842 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.822844 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.822860 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:32.822951 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.822957 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.824022 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.824025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.825371 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.825382 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826370 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.826471 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826506 (+ 35us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.826617 (+ 111us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.826728 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":107,"threads_started":1}
I20260811 04:40:32.826881 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.821702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.2.238.129:41513)
0811 04:40:32.822419 (+ 717us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.822430 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.822452 (+ 22us) server_negotiation.cc:407] Connection header received
0811 04:40:32.822520 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.822525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.822722 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.822866 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:32.823087 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.823849 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.825495 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826256 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826349 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826486 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.826644 (+ 158us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.826740 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":171,"threads_started":1}
I20260811 04:40:32.827047 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.820888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.2.238.130:43259)
0811 04:40:32.821683 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.821695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.821721 (+ 26us) server_negotiation.cc:407] Connection header received
0811 04:40:32.821825 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.821831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.822042 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.822226 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:32.822682 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.823827 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.825335 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826133 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826246 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826441 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.826800 (+ 359us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.826893 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":201,"threads_started":1}
I20260811 04:40:32.827318 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.821522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45177 (local address 127.0.0.1:42948)
0811 04:40:32.822363 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.822374 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.822489 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.823012 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.823017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.823036 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:32.823131 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.823139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.824461 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.824464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.825564 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.825585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826548 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.826653 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826668 (+ 15us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.826724 (+ 56us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.826806 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"mutex_wait_us":7,"thread_start_us":96,"threads_started":1}
I20260811 04:40:32.827596 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.820735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43259 (local address 127.0.0.1:47224)
0811 04:40:32.821103 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:32.821150 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:32.821272 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:32.822421 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:32.822428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:32.822450 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:32.822568 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.822574 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.823953 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.823958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.825208 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:32.825218 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826506 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:32.826598 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826653 (+ 55us) client_negotiation.cc:769] Sending connection context
0811 04:40:32.826723 (+ 70us) client_negotiation.cc:240] Negotiation successful
0811 04:40:32.827303 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"mutex_wait_us":568,"thread_start_us":106,"threads_started":1}
I20260811 04:40:32.828212 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:32.821960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42948 (local address 127.2.238.131:45177)
0811 04:40:32.822303 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:32.822313 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:32.822472 (+ 159us) server_negotiation.cc:407] Connection header received
0811 04:40:32.822601 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:32.822609 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:32.822811 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:32.822949 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:32.823250 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.824317 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.825703 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:32.826257 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:32.826369 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:32.826535 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:32.827931 (+ 1396us) server_negotiation.cc:299] Negotiation successful
0811 04:40:32.828041 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":159,"threads_started":1}
I20260811 04:40:32.830307 24426 tablet_service.cc:1467] Processing CreateTablet for tablet 619e8b6c32554affa6228729258bf50b (DEFAULT_TABLE table=test-table [id=f6d6b533d68f42b0bb3b446608f379ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:32.831256 24426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619e8b6c32554affa6228729258bf50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:32.831007 24715 tablet_service.cc:1467] Processing CreateTablet for tablet 619e8b6c32554affa6228729258bf50b (DEFAULT_TABLE table=test-table [id=f6d6b533d68f42b0bb3b446608f379ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:32.831002 24573 tablet_service.cc:1467] Processing CreateTablet for tablet 619e8b6c32554affa6228729258bf50b (DEFAULT_TABLE table=test-table [id=f6d6b533d68f42b0bb3b446608f379ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260811 04:40:32.832023 24573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619e8b6c32554affa6228729258bf50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:32.832023 24715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 619e8b6c32554affa6228729258bf50b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:32.837661 24805 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Bootstrap starting.
I20260811 04:40:32.839196 24807 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Bootstrap starting.
I20260811 04:40:32.839946 24806 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Bootstrap starting.
I20260811 04:40:32.840195 24805 tablet_bootstrap.cc:654] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:32.841092 24805 log.cc:824] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:32.841382 24807 tablet_bootstrap.cc:654] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:32.841964 24806 tablet_bootstrap.cc:654] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:32.842278 24807 log.cc:824] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:32.842784 24806 log.cc:824] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:32.844658 24807 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: No bootstrap required, opened a new log
I20260811 04:40:32.844658 24805 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: No bootstrap required, opened a new log
I20260811 04:40:32.844859 24805 ts_tablet_manager.cc:1397] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260811 04:40:32.844859 24807 ts_tablet_manager.cc:1397] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260811 04:40:32.845335 24806 tablet_bootstrap.cc:492] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: No bootstrap required, opened a new log
I20260811 04:40:32.845547 24806 ts_tablet_manager.cc:1397] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260811 04:40:32.848572 24806 raft_consensus.cc:348] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.848850 24806 raft_consensus.cc:374] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:32.848901 24806 raft_consensus.cc:729] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3062a1ffc844e42a11db1028f5e13ab, State: Initialized, Role: FOLLOWER
I20260811 04:40:32.849051 24805 raft_consensus.cc:348] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.849051 24807 raft_consensus.cc:348] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.849314 24807 raft_consensus.cc:374] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:32.849310 24805 raft_consensus.cc:374] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:32.849365 24807 raft_consensus.cc:729] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a09a6f26c2554201afbd69d9f3f3f9b5, State: Initialized, Role: FOLLOWER
I20260811 04:40:32.849375 24805 raft_consensus.cc:729] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835cbd1efa444d498e431ab41c70d798, State: Initialized, Role: FOLLOWER
I20260811 04:40:32.849351 24806 consensus_queue.cc:260] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.849828 24807 consensus_queue.cc:260] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.849843 24805 consensus_queue.cc:260] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:32.850430 24806 ts_tablet_manager.cc:1428] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
I20260811 04:40:32.850538 24645 heartbeater.cc:503] Master 127.2.238.190:42849 was elected leader, sending a full tablet report...
I20260811 04:40:32.850770 24787 heartbeater.cc:503] Master 127.2.238.190:42849 was elected leader, sending a full tablet report...
I20260811 04:40:32.850775 24805 ts_tablet_manager.cc:1428] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:40:32.850775 24807 ts_tablet_manager.cc:1428] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260811 04:40:32.851096 24502 heartbeater.cc:503] Master 127.2.238.190:42849 was elected leader, sending a full tablet report...
W20260811 04:40:32.904124 24503 tablet.cc:2369] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:33.036433 24788 tablet.cc:2369] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260811 04:40:33.090065 24646 tablet.cc:2369] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260811 04:40:33.109860 24814 raft_consensus.cc:482] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260811 04:40:33.110111 24814 raft_consensus.cc:504] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:33.111208 24814 leader_election.cc:288] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3062a1ffc844e42a11db1028f5e13ab (127.2.238.131:45177), 835cbd1efa444d498e431ab41c70d798 (127.2.238.129:41513)
I20260811 04:40:33.114687 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.111716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:33439 (local address 127.2.238.129:41513)
0811 04:40:33.111850 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:33.111855 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:33.111868 (+ 13us) server_negotiation.cc:407] Connection header received
0811 04:40:33.111922 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:33.111924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:33.111959 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:33.112035 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:33.112316 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.112761 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.113721 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.114124 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.114241 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:33.114292 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:33.114471 (+ 179us) server_negotiation.cc:299] Negotiation successful
0811 04:40:33.114541 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260811 04:40:33.114715 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.111505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.238.130:37241 (local address 127.2.238.131:45177)
0811 04:40:33.111743 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:33.111760 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:33.111774 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:40:33.111881 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:33.111884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:33.111921 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:33.111993 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0811 04:40:33.112276 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.112752 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.113726 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.114150 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.114271 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:33.114316 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:33.114512 (+ 196us) server_negotiation.cc:299] Negotiation successful
0811 04:40:33.114579 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260811 04:40:33.114737 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.111433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:41513 (local address 127.2.238.130:33439)
0811 04:40:33.111708 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:33.111721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:33.111820 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:33.112044 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:33.112046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:33.112077 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:33.112209 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:33.112214 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.112860 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.112864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:33.113603 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:33.113613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.114234 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.114238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:33.114333 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:33.114351 (+ 18us) client_negotiation.cc:769] Sending connection context
0811 04:40:33.114438 (+ 87us) client_negotiation.cc:240] Negotiation successful
0811 04:40:33.114524 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"thread_start_us":80,"threads_started":1}
I20260811 04:40:33.114923 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.111388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.131:45177 (local address 127.2.238.130:37241)
0811 04:40:33.111646 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:33.111663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0811 04:40:33.111772 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:40:33.112006 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:40:33.112008 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:40:33.112023 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0811 04:40:33.112173 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:33.112178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.112850 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.112854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:33.113611 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:40:33.113620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.114273 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.114276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:40:33.114374 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:33.114390 (+ 16us) client_negotiation.cc:769] Sending connection context
0811 04:40:33.114491 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 04:40:33.114565 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":36,"thread_start_us":80,"threads_started":1}
I20260811 04:40:33.115419 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619e8b6c32554affa6228729258bf50b" candidate_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835cbd1efa444d498e431ab41c70d798" is_pre_election: true
I20260811 04:40:33.115556 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619e8b6c32554affa6228729258bf50b" candidate_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3062a1ffc844e42a11db1028f5e13ab" is_pre_election: true
I20260811 04:40:33.115682 24449 raft_consensus.cc:2454] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09a6f26c2554201afbd69d9f3f3f9b5 in term 0.
I20260811 04:40:33.115850 24739 raft_consensus.cc:2454] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a09a6f26c2554201afbd69d9f3f3f9b5 in term 0.
I20260811 04:40:33.116153 24524 leader_election.cc:302] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835cbd1efa444d498e431ab41c70d798, a09a6f26c2554201afbd69d9f3f3f9b5; no voters:
I20260811 04:40:33.116436 24814 raft_consensus.cc:2781] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260811 04:40:33.116505 24814 raft_consensus.cc:482] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260811 04:40:33.116549 24814 raft_consensus.cc:3037] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:33.117475 24814 raft_consensus.cc:504] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:33.117831 24814 leader_election.cc:288] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [CANDIDATE]: Term 1 election: Requested vote from peers e3062a1ffc844e42a11db1028f5e13ab (127.2.238.131:45177), 835cbd1efa444d498e431ab41c70d798 (127.2.238.129:41513)
I20260811 04:40:33.118130 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619e8b6c32554affa6228729258bf50b" candidate_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3062a1ffc844e42a11db1028f5e13ab"
I20260811 04:40:33.118130 24449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "619e8b6c32554affa6228729258bf50b" candidate_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835cbd1efa444d498e431ab41c70d798"
I20260811 04:40:33.118242 24739 raft_consensus.cc:3037] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:33.118243 24449 raft_consensus.cc:3037] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:33.119423 24449 raft_consensus.cc:2454] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09a6f26c2554201afbd69d9f3f3f9b5 in term 1.
I20260811 04:40:33.119423 24739 raft_consensus.cc:2454] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a09a6f26c2554201afbd69d9f3f3f9b5 in term 1.
I20260811 04:40:33.119695 24524 leader_election.cc:302] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 835cbd1efa444d498e431ab41c70d798, a09a6f26c2554201afbd69d9f3f3f9b5, e3062a1ffc844e42a11db1028f5e13ab; no voters:
I20260811 04:40:33.119913 24814 raft_consensus.cc:2781] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:33.120093 24814 raft_consensus.cc:686] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 1 LEADER]: Becoming Leader. State: Replica: a09a6f26c2554201afbd69d9f3f3f9b5, State: Running, Role: LEADER
I20260811 04:40:33.120414 24814 consensus_queue.cc:237] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } }
I20260811 04:40:33.123338 24282 catalog_manager.cc:5495] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a09a6f26c2554201afbd69d9f3f3f9b5 (127.2.238.130). New cstate: current_term: 1 leader_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a09a6f26c2554201afbd69d9f3f3f9b5" member_type: VOTER last_known_addr { host: "127.2.238.130" port: 43259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 } health_report { overall_health: UNKNOWN } } }
I20260811 04:40:33.186714 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.183452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47234 (local address 127.2.238.130:43259)
0811 04:40:33.183589 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:33.183592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:33.183606 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:40:33.183763 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:33.183765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:33.183824 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:33.183912 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:40:33.184137 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.184628 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.185429 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:33.185641 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:33.185712 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:33.186029 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:40:33.186422 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:40:33.186509 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:33.186566 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 04:40:33.186609 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260811 04:40:33.192247 24739 raft_consensus.cc:1261] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 1 FOLLOWER]: Refusing update from remote peer a09a6f26c2554201afbd69d9f3f3f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:33.192261 24449 raft_consensus.cc:1261] T 619e8b6c32554affa6228729258bf50b P 835cbd1efa444d498e431ab41c70d798 [term 1 FOLLOWER]: Refusing update from remote peer a09a6f26c2554201afbd69d9f3f3f9b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:40:33.192919 24814 consensus_queue.cc:1035] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835cbd1efa444d498e431ab41c70d798" member_type: VOTER last_known_addr { host: "127.2.238.129" port: 41513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:33.193116 24818 consensus_queue.cc:1035] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3062a1ffc844e42a11db1028f5e13ab" member_type: VOTER last_known_addr { host: "127.2.238.131" port: 45177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:40:33.200734 24822 mvcc.cc:205] Tried to move back new op lower bound from 7317189563148447744 to 7317189562864001024. Current Snapshot: MvccSnapshot[applied={T|T < 7317189563148447744}]
I20260811 04:40:33.202229 24824 mvcc.cc:205] Tried to move back new op lower bound from 7317189563148447744 to 7317189562864001024. Current Snapshot: MvccSnapshot[applied={T|T < 7317189563148447744}]
I20260811 04:40:33.208592 24282 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:33.208721 24282 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260811 04:40:33.210846 24282 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb25bbf3b3c84f6b9d0d8ca287003586: Sending DeleteTablet for 3 replicas of tablet 619e8b6c32554affa6228729258bf50b
I20260811 04:40:33.211580 24426 tablet_service.cc:1514] Processing DeleteTablet for tablet 619e8b6c32554affa6228729258bf50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:33 UTC) from {username='slave'} at 127.0.0.1:50426
I20260811 04:40:33.211748 24715 tablet_service.cc:1514] Processing DeleteTablet for tablet 619e8b6c32554affa6228729258bf50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:33 UTC) from {username='slave'} at 127.0.0.1:42948
I20260811 04:40:33.211738 24573 tablet_service.cc:1514] Processing DeleteTablet for tablet 619e8b6c32554affa6228729258bf50b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-11 04:40:33 UTC) from {username='slave'} at 127.0.0.1:47224
I20260811 04:40:33.212096 24833 tablet_replica.cc:330] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: stopping tablet replica
I20260811 04:40:33.212293 24834 tablet_replica.cc:330] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5: stopping tablet replica
I20260811 04:40:33.212379 24833 raft_consensus.cc:2229] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260811 04:40:33.212571 24834 raft_consensus.cc:2229] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 1 LEADER]: Raft consensus shutting down.
I20260811 04:40:33.212636 24833 raft_consensus.cc:2258] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:33.212869 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24354
I20260811 04:40:33.212898 24834 raft_consensus.cc:2258] T 619e8b6c32554affa6228729258bf50b P a09a6f26c2554201afbd69d9f3f3f9b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260811 04:40:33.215016 24833 ts_tablet_manager.cc:1905] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260811 04:40:33.217746 24833 ts_tablet_manager.cc:1918] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260811 04:40:33.217872 24833 log.cc:1196] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Deleting WAL directory at /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786422982503306-3002-0/minicluster-data/ts-2/wal/wals/619e8b6c32554affa6228729258bf50b
I20260811 04:40:33.218194 24833 ts_tablet_manager.cc:1939] T 619e8b6c32554affa6228729258bf50b P e3062a1ffc844e42a11db1028f5e13ab: Deleting consensus metadata
I20260811 04:40:33.219501 24267 catalog_manager.cc:4841] TS e3062a1ffc844e42a11db1028f5e13ab (127.2.238.131:45177): tablet 619e8b6c32554affa6228729258bf50b (table test-table [id=f6d6b533d68f42b0bb3b446608f379ba]) successfully deleted
W20260811 04:40:33.221761 24268 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.238.129:41513 (error 108)
I20260811 04:40:33.222330 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24505
I20260811 04:40:33.224022 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.223705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.129:41513 (local address 127.0.0.1:50438)
0811 04:40:33.223853 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:33.223930 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.129:41513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260811 04:40:33.224628 24268 catalog_manager.cc:4584] TS 835cbd1efa444d498e431ab41c70d798 (127.2.238.129:41513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 619e8b6c32554affa6228729258bf50b: Network error: Client connection negotiation failed: client connection to 127.2.238.129:41513: connect: Connection refused (error 111)
I20260811 04:40:33.229683 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24648
I20260811 04:40:33.229983 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:33.229702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.130:43259 (local address 127.0.0.1:47236)
0811 04:40:33.229843 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0811 04:40:33.229891 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.130:43259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 04:40:33.236021 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24248
2026-08-11T04:40:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1291 ms)
[----------] 6 tests from EncryptionPolicyTest (10887 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:33 dist-test-slave-0kls krb5kdc[24839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:33 dist-test-slave-0kls krb5kdc[24839](info): set up 2 sockets
Aug 11 04:40:33 dist-test-slave-0kls krb5kdc[24839](info): commencing operation
krb5kdc: starting...
W20260811 04:40:34.278443 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:34.398976 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:34.508826 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:34.509068 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:34.509126 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:34.513055 24852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:34.513109 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:34.513128 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:34.513149 24852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:34.513207 24852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:34.518163 24852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:34.519083 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:34.520170 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:34.528482 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:34.528525 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:34.528684 24852 server_base.cc:1034] running on GCE node
W20260811 04:40:34.528891 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:34.529453 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:34.530586 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:34.531745 24852 hybrid_clock.cc:648] HybridClock initialized: now 1786423234531718 us; error 38 us; skew 500 ppm
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:34.537577 24852 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:34.538540 24852 webserver.cc:460] Webserver started at http://127.0.0.1:33649/ using document root <none> and password file <none>
I20260811 04:40:34.539088 24852 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:34.539177 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:34.539418 24852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:34.541476 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "767b4df741494cc1acf3b0c47f77bcfb"
format_stamp: "Formatted at 2026-08-11 04:40:34 on dist-test-slave-0kls"
server_key: "e64a46c38d4a1a41a69e19aa87b67ec8"
server_key_iv: "9b2c4cc0c372829b7e5fbea102b6c6de"
server_key_version: "encryptionkey@0"
I20260811 04:40:34.542130 24852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "767b4df741494cc1acf3b0c47f77bcfb"
format_stamp: "Formatted at 2026-08-11 04:40:34 on dist-test-slave-0kls"
server_key: "e64a46c38d4a1a41a69e19aa87b67ec8"
server_key_iv: "9b2c4cc0c372829b7e5fbea102b6c6de"
server_key_version: "encryptionkey@0"
I20260811 04:40:34.545873 24852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260811 04:40:34.548314 24868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:34.549484 24852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:40:34.549600 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "767b4df741494cc1acf3b0c47f77bcfb"
format_stamp: "Formatted at 2026-08-11 04:40:34 on dist-test-slave-0kls"
server_key: "e64a46c38d4a1a41a69e19aa87b67ec8"
server_key_iv: "9b2c4cc0c372829b7e5fbea102b6c6de"
server_key_version: "encryptionkey@0"
I20260811 04:40:34.549696 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:34.581634 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:34.585047 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:34.585217 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:34.592465 24852 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36695
I20260811 04:40:34.592494 24925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36695 every 8 connection(s)
I20260811 04:40:34.593621 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:34.596022 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24852
I20260811 04:40:34.596199 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:34.596650 3002 external_mini_cluster.cc:1428] Setting key cc606ce9a760306b8cb43380ad9c54e2
I20260811 04:40:34.596825 24926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:34.603446 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb: Bootstrap starting.
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 04:40:34.606043 24926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:34.606873 24926 log.cc:824] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:34.608743 24926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb: No bootstrap required, opened a new log
I20260811 04:40:34.611189 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:34.598185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34345 (local address 127.0.0.1:36695)
0811 04:40:34.598647 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:34.598663 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:34.598698 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:40:34.599597 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:34.599617 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:34.599974 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:34.600307 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:34.601148 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:34.602175 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:34.602955 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:34.603388 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:34.603549 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:34.606323 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:34.606328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:34.606340 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:34.606367 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:34.607976 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:34.608493 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:34.608497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:34.608501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:34.608598 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:34.608983 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:34.608986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:34.608988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:34.609476 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:34.609691 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:34.609913 (+ 222us) server_negotiation.cc:299] Negotiation successful
0811 04:40:34.610160 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":168,"threads_started":1}
I20260811 04:40:34.612879 24926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } }
I20260811 04:40:34.613135 24926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:34.613193 24926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 767b4df741494cc1acf3b0c47f77bcfb, State: Initialized, Role: FOLLOWER
I20260811 04:40:34.613595 24926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } }
I20260811 04:40:34.613803 24926 raft_consensus.cc:388] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:34.613871 24926 raft_consensus.cc:482] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:34.613965 24926 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:34.614251 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:34.615113 24926 raft_consensus.cc:504] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } }
I20260811 04:40:34.615482 24926 leader_election.cc:302] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 767b4df741494cc1acf3b0c47f77bcfb; no voters:
I20260811 04:40:34.615859 24926 leader_election.cc:288] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:34.615998 24935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:34.616284 24935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [term 1 LEADER]: Becoming Leader. State: Replica: 767b4df741494cc1acf3b0c47f77bcfb, State: Running, Role: LEADER
I20260811 04:40:34.616600 24935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } }
I20260811 04:40:34.617146 24926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:34.617749 24937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "767b4df741494cc1acf3b0c47f77bcfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } } }
I20260811 04:40:34.617749 24935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 767b4df741494cc1acf3b0c47f77bcfb. Latest consensus state: current_term: 1 leader_uuid: "767b4df741494cc1acf3b0c47f77bcfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "767b4df741494cc1acf3b0c47f77bcfb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36695 } } }
I20260811 04:40:34.617929 24937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:34.617929 24935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:34.618310 24941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:34.621524 24941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:34.626411 24929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:34.616845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58499 (local address 127.0.0.1:36695)
0811 04:40:34.616935 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:34.616939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:34.616951 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:40:34.617068 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:34.617071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:34.617111 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:34.617159 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:34.618418 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:34.619193 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:34.620403 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:34.620894 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:34.621027 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:34.622740 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:34.622743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:34.622746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:34.622770 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:34.624332 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:34.624896 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:34.624899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:34.624901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:34.624991 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:34.625407 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:34.625410 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:34.625412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:34.625663 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:34.625904 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:34.626175 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 04:40:34.626280 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 04:40:34.627540 24941 catalog_manager.cc:1347] Generated new cluster ID: 638d950737cb44eaa159c83c65698626
I20260811 04:40:34.627599 24941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:34.648885 24941 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:34.649935 24941 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:34.663316 24941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 767b4df741494cc1acf3b0c47f77bcfb: Generated new TSK 0
I20260811 04:40:34.663985 24941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:34.677793 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24852
W20260811 04:40:34.685456 24837 connection.cc:535] client connection to 127.0.0.1:36695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36695): Connection reset by peer (error 104)
2026-08-11T04:40:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24960](info): set up 2 sockets
Aug 11 04:40:34 dist-test-slave-0kls krb5kdc[24960](info): commencing operation
krb5kdc: starting...
W20260811 04:40:35.721258 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:35 dist-test-slave-0kls krb5kdc[24960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:35.844645 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:35.960685 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:35.960924 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:35.960964 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:35.964825 24975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:35.964875 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:35.964893 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:35.964910 24975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:35.964924 24975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:35.969692 24975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34833
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:35.970672 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:35.971606 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:35.979568 24980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:35.979681 24975 server_base.cc:1034] running on GCE node
W20260811 04:40:35.979630 24981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:35.979617 24983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:35.980499 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:35.981467 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:35.982637 24975 hybrid_clock.cc:648] HybridClock initialized: now 1786423235982607 us; error 41 us; skew 500 ppm
Aug 11 04:40:35 dist-test-slave-0kls krb5kdc[24960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:35.985175 24975 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:35.986066 24975 webserver.cc:460] Webserver started at http://127.0.0.1:34713/ using document root <none> and password file <none>
I20260811 04:40:35.986526 24975 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:35.986577 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:35.986726 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:35.989179 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "9b49aabc1bcb4a66bf1799348be512ec"
format_stamp: "Formatted at 2026-08-11 04:40:35 on dist-test-slave-0kls"
server_key: "e5bda789cfa0613111b8112ac3af7864"
server_key_iv: "cd7ee758c4ae622391b80fcfe4b3c381"
server_key_version: "encryptionkey@0"
I20260811 04:40:35.989859 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "9b49aabc1bcb4a66bf1799348be512ec"
format_stamp: "Formatted at 2026-08-11 04:40:35 on dist-test-slave-0kls"
server_key: "e5bda789cfa0613111b8112ac3af7864"
server_key_iv: "cd7ee758c4ae622391b80fcfe4b3c381"
server_key_version: "encryptionkey@0"
I20260811 04:40:35.994719 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260811 04:40:35.997323 24991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:35.998600 24975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:40:35.998752 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "9b49aabc1bcb4a66bf1799348be512ec"
format_stamp: "Formatted at 2026-08-11 04:40:35 on dist-test-slave-0kls"
server_key: "e5bda789cfa0613111b8112ac3af7864"
server_key_iv: "cd7ee758c4ae622391b80fcfe4b3c381"
server_key_version: "encryptionkey@0"
I20260811 04:40:35.998867 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:36.020078 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:36.023497 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:36.023710 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:36.030882 24975 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34833
I20260811 04:40:36.030910 25044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34833 every 8 connection(s)
I20260811 04:40:36.031805 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:36.034742 25045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:36.040244 25045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec: Bootstrap starting.
I20260811 04:40:36.041286 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 24975
I20260811 04:40:36.041430 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:36.041754 3002 external_mini_cluster.cc:1428] Setting key cf978da3e58a4b1b3b923b00e985524e
I20260811 04:40:36.043251 25045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:36.044010 25045 log.cc:824] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:36.046113 25045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec: No bootstrap required, opened a new log
I20260811 04:40:36.050508 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:36.042857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56593 (local address 127.0.0.1:34833)
0811 04:40:36.043285 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:36.043311 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:36.043366 (+ 55us) server_negotiation.cc:407] Connection header received
0811 04:40:36.044013 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:36.044029 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:36.044457 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:36.044835 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:36.045164 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:36.046177 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:36.047372 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:36.047828 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:36.047940 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:36.048844 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:36.048849 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:36.048861 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:36.049223 (+ 362us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:36.049275 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:36.049295 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:36.049394 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:36.049626 (+ 232us) server_negotiation.cc:299] Negotiation successful
0811 04:40:36.049771 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":158,"threads_started":1}
I20260811 04:40:36.050432 25045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260811 04:40:36.050707 25045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:36.050796 25045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b49aabc1bcb4a66bf1799348be512ec, State: Initialized, Role: FOLLOWER
I20260811 04:40:36.051277 25045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260811 04:40:36.051431 25045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:36.051517 25045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:36.051617 25045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:36.052727 25045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260811 04:40:36.053105 25045 leader_election.cc:302] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b49aabc1bcb4a66bf1799348be512ec; no voters:
I20260811 04:40:36.053463 25045 leader_election.cc:288] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:36.053579 25051 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:36.053630 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:36.053835 25051 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [term 1 LEADER]: Becoming Leader. State: Replica: 9b49aabc1bcb4a66bf1799348be512ec, State: Running, Role: LEADER
I20260811 04:40:36.054134 25051 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } }
I20260811 04:40:36.054836 25045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:36.055641 25052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b49aabc1bcb4a66bf1799348be512ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } } }
I20260811 04:40:36.055760 25052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:36.056138 25053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b49aabc1bcb4a66bf1799348be512ec. Latest consensus state: current_term: 1 leader_uuid: "9b49aabc1bcb4a66bf1799348be512ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b49aabc1bcb4a66bf1799348be512ec" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34833 } } }
I20260811 04:40:36.056282 25053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:36.056599 25064 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:36.059634 25064 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:36.061287 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:36.056131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40585 (local address 127.0.0.1:34833)
0811 04:40:36.056239 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:36.056243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:36.057282 (+ 1039us) server_negotiation.cc:407] Connection header received
0811 04:40:36.057414 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:36.057418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:36.057464 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:36.057549 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:36.057889 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:36.058390 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:36.059498 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:36.059782 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:36.059925 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:36.060804 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:36.060807 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:36.060810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:36.060822 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:36.060855 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:36.060866 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:36.060963 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:36.061108 (+ 145us) server_negotiation.cc:299] Negotiation successful
0811 04:40:36.061141 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:40:36.065553 25064 catalog_manager.cc:1347] Generated new cluster ID: a7e860d1144e4a9f960a1e6a0ccf6dca
I20260811 04:40:36.065629 25064 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:36.080302 25064 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:36.081436 25064 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:36.089145 25064 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b49aabc1bcb4a66bf1799348be512ec: Generated new TSK 0
I20260811 04:40:36.089804 25064 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:36.095537 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 24975
W20260811 04:40:36.103427 24958 connection.cc:535] client connection to 127.0.0.1:34833 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34833): Connection reset by peer (error 104)
2026-08-11T04:40:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:36 dist-test-slave-0kls krb5kdc[25078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:36 dist-test-slave-0kls krb5kdc[25078](info): set up 2 sockets
Aug 11 04:40:36 dist-test-slave-0kls krb5kdc[25078](info): commencing operation
krb5kdc: starting...
W20260811 04:40:37.136808 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:37 dist-test-slave-0kls krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:37.258358 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:37.366118 25092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:37.366389 25092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:37.366438 25092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:37.370007 25092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:37.370101 25092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:37.370163 25092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:37.370225 25092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:37.370265 25092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:37.375113 25092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44299
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:37.376132 25092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:37.377166 25092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:37.385416 25099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:37.385464 25102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:37.385571 25098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:37.385653 25092 server_base.cc:1034] running on GCE node
I20260811 04:40:37.386303 25092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:37.387395 25092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:37.388581 25092 hybrid_clock.cc:648] HybridClock initialized: now 1786423237388563 us; error 27 us; skew 500 ppm
Aug 11 04:40:37 dist-test-slave-0kls krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:37.391397 25092 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:37.392331 25092 webserver.cc:460] Webserver started at http://127.0.0.1:34089/ using document root <none> and password file <none>
I20260811 04:40:37.392875 25092 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:37.392967 25092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:37.393204 25092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:37.395277 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "1fce15db31e849dcb1d2f3ee4b126431"
format_stamp: "Formatted at 2026-08-11 04:40:37 on dist-test-slave-0kls"
server_key: "4588d11c9a0d6ec44206b4aa1aacd7fa"
server_key_iv: "673a165ce2974fb2bfd748163c1530fc"
server_key_version: "encryptionkey@0"
I20260811 04:40:37.395844 25092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "1fce15db31e849dcb1d2f3ee4b126431"
format_stamp: "Formatted at 2026-08-11 04:40:37 on dist-test-slave-0kls"
server_key: "4588d11c9a0d6ec44206b4aa1aacd7fa"
server_key_iv: "673a165ce2974fb2bfd748163c1530fc"
server_key_version: "encryptionkey@0"
I20260811 04:40:37.399389 25092 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:40:37.401703 25111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:37.402842 25092 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:37.402976 25092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "1fce15db31e849dcb1d2f3ee4b126431"
format_stamp: "Formatted at 2026-08-11 04:40:37 on dist-test-slave-0kls"
server_key: "4588d11c9a0d6ec44206b4aa1aacd7fa"
server_key_iv: "673a165ce2974fb2bfd748163c1530fc"
server_key_version: "encryptionkey@0"
I20260811 04:40:37.403085 25092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:37.407509 25092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:37.410521 25092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:37.410699 25092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:37.417514 25092 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44299
I20260811 04:40:37.417537 25166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44299 every 8 connection(s)
I20260811 04:40:37.418387 25092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:37.421041 25167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:37.423851 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25092
I20260811 04:40:37.424022 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:37.424304 3002 external_mini_cluster.cc:1428] Setting key 6fa2fb36b02744ee682c9e803086fdd0
I20260811 04:40:37.426673 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431: Bootstrap starting.
I20260811 04:40:37.428881 25167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:37.429096 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:37.425593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37345 (local address 127.0.0.1:44299)
0811 04:40:37.426074 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:37.426088 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:37.426119 (+ 31us) server_negotiation.cc:407] Connection header received
0811 04:40:37.426667 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:37.426681 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:37.426988 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:37.427282 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:37.428015 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:37.428019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:37.428030 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:37.428039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:37.428088 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:37.428106 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:37.428248 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:37.428313 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 04:40:37.428456 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":136,"threads_started":1}
I20260811 04:40:37.429687 25167 log.cc:824] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:37.431833 25167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431: No bootstrap required, opened a new log
I20260811 04:40:37.431937 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:37.435432 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:37.433849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51733 (local address 127.0.0.1:44299)
0811 04:40:37.433966 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:37.433970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:37.434182 (+ 212us) server_negotiation.cc:407] Connection header received
0811 04:40:37.434293 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:37.434295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:37.434319 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:37.434388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:37.435071 (+ 683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:37.435074 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:37.435075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:37.435085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:37.435107 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:37.435117 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:37.435185 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:37.435283 (+ 98us) server_negotiation.cc:299] Negotiation successful
0811 04:40:37.435310 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:40:37.435855 25167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260811 04:40:37.436159 25167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:37.436270 25167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fce15db31e849dcb1d2f3ee4b126431, State: Initialized, Role: FOLLOWER
I20260811 04:40:37.436710 25167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260811 04:40:37.436864 25167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:37.436928 25167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:37.437018 25167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:37.437885 25167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260811 04:40:37.438197 25167 leader_election.cc:302] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fce15db31e849dcb1d2f3ee4b126431; no voters:
I20260811 04:40:37.438463 25167 leader_election.cc:288] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:37.438568 25178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:37.438832 25178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [term 1 LEADER]: Becoming Leader. State: Replica: 1fce15db31e849dcb1d2f3ee4b126431, State: Running, Role: LEADER
I20260811 04:40:37.439100 25178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } }
I20260811 04:40:37.439527 25167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:37.440892 25179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fce15db31e849dcb1d2f3ee4b126431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260811 04:40:37.440893 25180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fce15db31e849dcb1d2f3ee4b126431. Latest consensus state: current_term: 1 leader_uuid: "1fce15db31e849dcb1d2f3ee4b126431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fce15db31e849dcb1d2f3ee4b126431" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44299 } } }
I20260811 04:40:37.441042 25179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:37.441097 25180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:37.441519 25189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:37.444003 25189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:37.449419 25189 catalog_manager.cc:1347] Generated new cluster ID: ea7b36c601454efa8e0b4b777ef8b9c2
I20260811 04:40:37.449491 25189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:37.471002 25189 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:37.471334 25189 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:37.479805 25189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fce15db31e849dcb1d2f3ee4b126431: Generated new TSK 0
I20260811 04:40:37.480521 25189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:37.487254 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25092
2026-08-11T04:40:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:37 dist-test-slave-0kls krb5kdc[25202](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:37 dist-test-slave-0kls krb5kdc[25202](info): set up 2 sockets
Aug 11 04:40:37 dist-test-slave-0kls krb5kdc[25202](info): commencing operation
krb5kdc: starting...
W20260811 04:40:38.541356 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:38.664575 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34503
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34503
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:38.772298 25215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:38.772594 25215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:38.772683 25215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:38.776670 25215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:38.776755 25215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:38.776793 25215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:38.776852 25215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:38.776888 25215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:38.781919 25215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34503
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34503
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:38.782910 25215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:38.783905 25215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:38.792042 25221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:38.792048 25224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:38.792097 25222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:38.792295 25215 server_base.cc:1034] running on GCE node
I20260811 04:40:38.792906 25215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:38.793999 25215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:38.795189 25215 hybrid_clock.cc:648] HybridClock initialized: now 1786423238795150 us; error 49 us; skew 500 ppm
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25202](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:38.797951 25215 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:38.798866 25215 webserver.cc:460] Webserver started at http://127.0.0.1:43819/ using document root <none> and password file <none>
I20260811 04:40:38.799425 25215 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:38.799517 25215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:38.799746 25215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:38.801913 25215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "07a6d5ec06224634a46e0a61a0b73573"
format_stamp: "Formatted at 2026-08-11 04:40:38 on dist-test-slave-0kls"
server_key: "40e7d9484beab05e57b3b34a236ba4d1"
server_key_iv: "cd601c7e6c7fe5f55c9ec512e2a102d9"
server_key_version: "encryptionkey@0"
I20260811 04:40:38.802501 25215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "07a6d5ec06224634a46e0a61a0b73573"
format_stamp: "Formatted at 2026-08-11 04:40:38 on dist-test-slave-0kls"
server_key: "40e7d9484beab05e57b3b34a236ba4d1"
server_key_iv: "cd601c7e6c7fe5f55c9ec512e2a102d9"
server_key_version: "encryptionkey@0"
I20260811 04:40:38.806264 25215 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:38.808787 25232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:38.810037 25215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:38.810197 25215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "07a6d5ec06224634a46e0a61a0b73573"
format_stamp: "Formatted at 2026-08-11 04:40:38 on dist-test-slave-0kls"
server_key: "40e7d9484beab05e57b3b34a236ba4d1"
server_key_iv: "cd601c7e6c7fe5f55c9ec512e2a102d9"
server_key_version: "encryptionkey@0"
I20260811 04:40:38.810284 25215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:38.829625 25215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:38.832929 25215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:38.833091 25215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:38.840178 25215 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34503
I20260811 04:40:38.840188 25291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34503 every 8 connection(s)
I20260811 04:40:38.841128 25215 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:38.843968 25292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:38.849255 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573: Bootstrap starting.
I20260811 04:40:38.850842 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25215
I20260811 04:40:38.851006 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:38.851280 3002 external_mini_cluster.cc:1428] Setting key 6acdf36261c09a747d99996009418efb
I20260811 04:40:38.851943 25292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:38.852787 25292 log.cc:824] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:38.854914 25292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573: No bootstrap required, opened a new log
I20260811 04:40:38.859428 25292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } }
I20260811 04:40:38.859681 25292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:38.859760 25292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a6d5ec06224634a46e0a61a0b73573, State: Initialized, Role: FOLLOWER
I20260811 04:40:38.860234 25292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } }
I20260811 04:40:38.860402 25292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:38.860462 25292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:38.860572 25292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 0 FOLLOWER]: Advancing to term 1
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25202](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 04:40:38.861645 25292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } }
I20260811 04:40:38.862040 25292 leader_election.cc:302] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a6d5ec06224634a46e0a61a0b73573; no voters:
I20260811 04:40:38.862401 25292 leader_election.cc:288] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:38.862495 25298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:38.862731 25298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [term 1 LEADER]: Becoming Leader. State: Replica: 07a6d5ec06224634a46e0a61a0b73573, State: Running, Role: LEADER
I20260811 04:40:38.863112 25298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } }
I20260811 04:40:38.863644 25292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:38.864353 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a6d5ec06224634a46e0a61a0b73573. Latest consensus state: current_term: 1 leader_uuid: "07a6d5ec06224634a46e0a61a0b73573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } } }
I20260811 04:40:38.864547 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:38.864993 25309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:38.866582 25300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a6d5ec06224634a46e0a61a0b73573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6d5ec06224634a46e0a61a0b73573" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34503 } } }
I20260811 04:40:38.866688 25300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:38.867348 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:38.852429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59897 (local address 127.0.0.1:34503)
0811 04:40:38.852872 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:38.852891 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:38.852930 (+ 39us) server_negotiation.cc:407] Connection header received
0811 04:40:38.853701 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:38.853721 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:38.854110 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:38.854432 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:38.855151 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:38.856616 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:38.857598 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:38.858442 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:38.858635 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:38.861427 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:38.861432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:38.861445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:38.861471 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:38.863465 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:38.863974 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:38.863979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:38.863984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:38.864136 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:38.864473 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:38.864478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:38.864482 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:38.865061 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:38.865953 (+ 892us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:38.866122 (+ 169us) server_negotiation.cc:299] Negotiation successful
0811 04:40:38.866431 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":125,"threads_started":1}
I20260811 04:40:38.869108 25309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:38.870410 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:38.875778 25309 catalog_manager.cc:1347] Generated new cluster ID: dcf28b1496784bf9beca9a80adad2796
I20260811 04:40:38.875836 25309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:38.882563 25296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:38.872388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54975 (local address 127.0.0.1:34503)
0811 04:40:38.872496 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:38.872500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:38.874062 (+ 1562us) server_negotiation.cc:407] Connection header received
0811 04:40:38.874212 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:38.874218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:38.874271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:38.874352 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:38.875328 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:38.876065 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:38.877174 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:38.877549 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:38.877623 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:38.879295 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:38.879299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:38.879301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:38.879323 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:38.880939 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:38.881372 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:38.881375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:38.881377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:38.881460 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:38.881726 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:38.881729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:38.881730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:38.881925 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:38.882016 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:38.882313 (+ 297us) server_negotiation.cc:299] Negotiation successful
0811 04:40:38.882413 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260811 04:40:38.907332 25309 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:38.908301 25309 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:38.913616 25309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a6d5ec06224634a46e0a61a0b73573: Generated new TSK 0
I20260811 04:40:38.914321 25309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:38.918916 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25215
2026-08-11T04:40:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25327](info): set up 2 sockets
Aug 11 04:40:38 dist-test-slave-0kls krb5kdc[25327](info): commencing operation
krb5kdc: starting...
W20260811 04:40:40.002117 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:40 dist-test-slave-0kls krb5kdc[25327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:40.132731 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:40.240607 25342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:40.240849 25342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:40.240888 25342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:40.244933 25342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:40.244985 25342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:40.245005 25342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:40.245021 25342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:40.245036 25342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:40.250133 25342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:40.251173 25342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:40.252177 25342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:40.260903 25350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:40.260871 25348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:40.261008 25342 server_base.cc:1034] running on GCE node
W20260811 04:40:40.260910 25347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:40.261745 25342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:40.262910 25342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:40.264086 25342 hybrid_clock.cc:648] HybridClock initialized: now 1786423240264057 us; error 38 us; skew 500 ppm
Aug 11 04:40:40 dist-test-slave-0kls krb5kdc[25327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:40.293408 25342 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:40.294492 25342 webserver.cc:460] Webserver started at http://127.0.0.1:43541/ using document root <none> and password file <none>
I20260811 04:40:40.295038 25342 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:40.295097 25342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:40.295279 25342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:40.297308 25342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "3b21a05773594ddcb02b4fcbeec53ec4"
format_stamp: "Formatted at 2026-08-11 04:40:40 on dist-test-slave-0kls"
server_key: "be392af9b0952c732dcd38d127b11a19"
server_key_iv: "463252f1d2dd415038b68f682db7898d"
server_key_version: "encryptionkey@0"
I20260811 04:40:40.297904 25342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "3b21a05773594ddcb02b4fcbeec53ec4"
format_stamp: "Formatted at 2026-08-11 04:40:40 on dist-test-slave-0kls"
server_key: "be392af9b0952c732dcd38d127b11a19"
server_key_iv: "463252f1d2dd415038b68f682db7898d"
server_key_version: "encryptionkey@0"
I20260811 04:40:40.301563 25342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260811 04:40:40.304000 25359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:40.305279 25342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:40.305379 25342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "3b21a05773594ddcb02b4fcbeec53ec4"
format_stamp: "Formatted at 2026-08-11 04:40:40 on dist-test-slave-0kls"
server_key: "be392af9b0952c732dcd38d127b11a19"
server_key_iv: "463252f1d2dd415038b68f682db7898d"
server_key_version: "encryptionkey@0"
I20260811 04:40:40.305462 25342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:40.315052 25342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:40.317720 25342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:40.317927 25342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:40.325052 25342 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45233
I20260811 04:40:40.325078 25415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45233 every 8 connection(s)
I20260811 04:40:40.326033 25342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:40.328881 25416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:40.329067 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25342
I20260811 04:40:40.329207 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:40.329510 3002 external_mini_cluster.cc:1428] Setting key 941300d39abf065907e712fb0d9b3033
I20260811 04:40:40.335304 25416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4: Bootstrap starting.
I20260811 04:40:40.338027 25416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:40.338625 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:40.330924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39535 (local address 127.0.0.1:45233)
0811 04:40:40.331345 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:40.331372 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:40.331422 (+ 50us) server_negotiation.cc:407] Connection header received
0811 04:40:40.332191 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:40.332207 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:40.332525 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:40.332856 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:40.333239 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:40.334242 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:40.335136 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:40.335658 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:40.335833 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:40.336805 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:40.336811 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:40.336832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:40.337303 (+ 471us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:40.337358 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:40.337381 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:40.337487 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:40.337607 (+ 120us) server_negotiation.cc:299] Negotiation successful
0811 04:40:40.337844 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260811 04:40:40.338837 25416 log.cc:824] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:40.340912 25416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4: No bootstrap required, opened a new log
I20260811 04:40:40.341382 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:40.344817 25416 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260811 04:40:40.345096 25416 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:40.345165 25416 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b21a05773594ddcb02b4fcbeec53ec4, State: Initialized, Role: FOLLOWER
I20260811 04:40:40.345595 25416 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260811 04:40:40.345791 25416 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:40.345906 25416 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:40.346029 25416 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:40.347263 25416 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260811 04:40:40.347445 25420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:40.343322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43715 (local address 127.0.0.1:45233)
0811 04:40:40.343427 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:40.343438 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:40.343632 (+ 194us) server_negotiation.cc:407] Connection header received
0811 04:40:40.343767 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:40.343769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:40.343822 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:40.343879 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:40.344175 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:40.344623 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:40.345712 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:40.346162 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:40.346260 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:40.347004 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:40.347007 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:40.347009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:40.347019 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:40.347044 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:40.347054 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:40.347111 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:40.347262 (+ 151us) server_negotiation.cc:299] Negotiation successful
0811 04:40:40.347305 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:40:40.347649 25416 leader_election.cc:302] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b21a05773594ddcb02b4fcbeec53ec4; no voters:
I20260811 04:40:40.348039 25416 leader_election.cc:288] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:40.348126 25427 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:40.348567 25427 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [term 1 LEADER]: Becoming Leader. State: Replica: 3b21a05773594ddcb02b4fcbeec53ec4, State: Running, Role: LEADER
I20260811 04:40:40.348876 25427 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260811 04:40:40.349257 25416 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:40.350471 25429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260811 04:40:40.350597 25429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:40.350971 25428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b21a05773594ddcb02b4fcbeec53ec4. Latest consensus state: current_term: 1 leader_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b21a05773594ddcb02b4fcbeec53ec4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260811 04:40:40.351049 25438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:40.354130 25438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:40.351090 25428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:40.359337 25438 catalog_manager.cc:1347] Generated new cluster ID: 5c7fc11da3834e6984c5e77dbca1c971
I20260811 04:40:40.359396 25438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:40.388440 25438 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:40.389447 25438 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:40.398233 25438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b21a05773594ddcb02b4fcbeec53ec4: Generated new TSK 0
I20260811 04:40:40.399058 25438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:40.414273 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25342
2026-08-11T04:40:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1494 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:40 dist-test-slave-0kls krb5kdc[25451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:40 dist-test-slave-0kls krb5kdc[25451](info): set up 2 sockets
Aug 11 04:40:40 dist-test-slave-0kls krb5kdc[25451](info): commencing operation
krb5kdc: starting...
W20260811 04:40:41.494405 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:41 dist-test-slave-0kls krb5kdc[25451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:41.620327 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:41.734824 25466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:41.735122 25466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:41.735165 25466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:41.739374 25466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:41.739431 25466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:41.739451 25466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:41.739468 25466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:41.739483 25466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:41.744701 25466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:41.745703 25466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:41.746848 25466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:41.755467 25472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:41.755599 25466 server_base.cc:1034] running on GCE node
W20260811 04:40:41.755435 25474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:41.755429 25471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:41.756328 25466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:41.757421 25466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:41.758605 25466 hybrid_clock.cc:648] HybridClock initialized: now 1786423241758579 us; error 33 us; skew 500 ppm
Aug 11 04:40:41 dist-test-slave-0kls krb5kdc[25451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:41.761663 25466 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:41.762571 25466 webserver.cc:460] Webserver started at http://127.0.0.1:40887/ using document root <none> and password file <none>
I20260811 04:40:41.763136 25466 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:41.763229 25466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:41.763521 25466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:41.765588 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "fe1ccc477e064bc9901e4cb2a564489c"
format_stamp: "Formatted at 2026-08-11 04:40:41 on dist-test-slave-0kls"
server_key: "56ccb05b239f31ff1f5c670c472dd8bc"
server_key_iv: "4b47783e58380bfea4a712bad9c2731f"
server_key_version: "encryptionkey@0"
I20260811 04:40:41.766232 25466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "fe1ccc477e064bc9901e4cb2a564489c"
format_stamp: "Formatted at 2026-08-11 04:40:41 on dist-test-slave-0kls"
server_key: "56ccb05b239f31ff1f5c670c472dd8bc"
server_key_iv: "4b47783e58380bfea4a712bad9c2731f"
server_key_version: "encryptionkey@0"
I20260811 04:40:41.769956 25466 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:41.772295 25481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:41.773474 25466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:41.773627 25466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "fe1ccc477e064bc9901e4cb2a564489c"
format_stamp: "Formatted at 2026-08-11 04:40:41 on dist-test-slave-0kls"
server_key: "56ccb05b239f31ff1f5c670c472dd8bc"
server_key_iv: "4b47783e58380bfea4a712bad9c2731f"
server_key_version: "encryptionkey@0"
I20260811 04:40:41.773738 25466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:41.778384 25466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:41.781246 25466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:41.781384 25466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:41.788141 25466 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37129
I20260811 04:40:41.788200 25536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37129 every 8 connection(s)
I20260811 04:40:41.789044 25466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:41.791818 25537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:41.797081 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25466
I20260811 04:40:41.797237 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:41.797473 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c: Bootstrap starting.
I20260811 04:40:41.797569 3002 external_mini_cluster.cc:1428] Setting key 7ce69a7109b51bd535764d266d07f296
I20260811 04:40:41.800011 25537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:41.800876 25537 log.cc:824] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:41.802757 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:41.798843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53771 (local address 127.0.0.1:37129)
0811 04:40:41.799246 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:41.799270 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:41.799305 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:40:41.799930 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:41.799946 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:41.800268 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:41.800538 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:41.801340 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:41.801350 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:41.801361 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:41.801372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:41.801422 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:41.801440 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:41.801525 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:41.801717 (+ 192us) server_negotiation.cc:299] Negotiation successful
0811 04:40:41.801896 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":124,"threads_started":1}
I20260811 04:40:41.803138 25537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c: No bootstrap required, opened a new log
I20260811 04:40:41.805480 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:41.807140 25537 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260811 04:40:41.807341 25537 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:41.807412 25537 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe1ccc477e064bc9901e4cb2a564489c, State: Initialized, Role: FOLLOWER
I20260811 04:40:41.807901 25537 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260811 04:40:41.808087 25537 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:41.808171 25537 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:41.808284 25537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:41.809469 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:41.807487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54009 (local address 127.0.0.1:37129)
0811 04:40:41.807602 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:41.807608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:41.808097 (+ 489us) server_negotiation.cc:407] Connection header received
0811 04:40:41.808201 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:41.808205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:41.808273 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:41.808398 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:41.809110 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:41.809113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:41.809115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:41.809126 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:41.809152 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:41.809163 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:41.809240 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:41.809355 (+ 115us) server_negotiation.cc:299] Negotiation successful
0811 04:40:41.809385 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260811 04:40:41.809481 25537 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260811 04:40:41.809903 25537 leader_election.cc:302] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe1ccc477e064bc9901e4cb2a564489c; no voters:
I20260811 04:40:41.810257 25537 leader_election.cc:288] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:41.810340 25549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:41.810554 25549 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [term 1 LEADER]: Becoming Leader. State: Replica: fe1ccc477e064bc9901e4cb2a564489c, State: Running, Role: LEADER
I20260811 04:40:41.810839 25549 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } }
I20260811 04:40:41.811529 25537 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:41.811967 25549 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe1ccc477e064bc9901e4cb2a564489c. Latest consensus state: current_term: 1 leader_uuid: "fe1ccc477e064bc9901e4cb2a564489c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } } }
I20260811 04:40:41.812088 25549 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:41.812448 25550 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe1ccc477e064bc9901e4cb2a564489c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe1ccc477e064bc9901e4cb2a564489c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37129 } } }
I20260811 04:40:41.812577 25550 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:41.812873 25553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:41.815925 25553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:41.821508 25553 catalog_manager.cc:1347] Generated new cluster ID: 693969078b5941a692a4a6632143ad95
I20260811 04:40:41.821584 25553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:41.840406 25553 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:41.840911 25553 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:41.847826 25553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe1ccc477e064bc9901e4cb2a564489c: Generated new TSK 0
I20260811 04:40:41.848538 25553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:41.852936 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25466
2026-08-11T04:40:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1441 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:41 dist-test-slave-0kls krb5kdc[25572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:41 dist-test-slave-0kls krb5kdc[25572](info): set up 2 sockets
Aug 11 04:40:41 dist-test-slave-0kls krb5kdc[25572](info): commencing operation
krb5kdc: starting...
W20260811 04:40:42.922923 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:43.046036 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42149
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:43.153575 25586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:43.153824 25586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:43.153863 25586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:43.157859 25586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:43.157909 25586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:43.157928 25586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:43.157944 25586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:43.157989 25586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:43.162941 25586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42149
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:43.163870 25586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:43.164821 25586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:40:43.173260 25586 server_base.cc:1034] running on GCE node
W20260811 04:40:43.173342 25593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:43.173298 25595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:43.173317 25592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:43.174140 25586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:43.175110 25586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:43.176331 25586 hybrid_clock.cc:648] HybridClock initialized: now 1786423243176300 us; error 28 us; skew 500 ppm
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:43.178987 25586 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:43.179836 25586 webserver.cc:460] Webserver started at http://127.0.0.1:40073/ using document root <none> and password file <none>
I20260811 04:40:43.180310 25586 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:43.180364 25586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:43.180516 25586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:43.182497 25586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "dbb49a2ce38643c0a9203880e2f3c135"
format_stamp: "Formatted at 2026-08-11 04:40:43 on dist-test-slave-0kls"
server_key: "58bb2a4f3d62216c0835517f5bb8ff57"
server_key_iv: "00f9a5d8c4efac80fb0de44edeaf74a0"
server_key_version: "encryptionkey@0"
I20260811 04:40:43.183005 25586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "dbb49a2ce38643c0a9203880e2f3c135"
format_stamp: "Formatted at 2026-08-11 04:40:43 on dist-test-slave-0kls"
server_key: "58bb2a4f3d62216c0835517f5bb8ff57"
server_key_iv: "00f9a5d8c4efac80fb0de44edeaf74a0"
server_key_version: "encryptionkey@0"
I20260811 04:40:43.186458 25586 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:43.188573 25603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:43.189728 25586 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:43.189911 25586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "dbb49a2ce38643c0a9203880e2f3c135"
format_stamp: "Formatted at 2026-08-11 04:40:43 on dist-test-slave-0kls"
server_key: "58bb2a4f3d62216c0835517f5bb8ff57"
server_key_iv: "00f9a5d8c4efac80fb0de44edeaf74a0"
server_key_version: "encryptionkey@0"
I20260811 04:40:43.190011 25586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:43.208148 25586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:43.211094 25586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:43.211299 25586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:43.218858 25586 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42149
I20260811 04:40:43.218884 25658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42149 every 8 connection(s)
I20260811 04:40:43.219900 25586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:43.222000 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25586
I20260811 04:40:43.222131 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:43.222441 3002 external_mini_cluster.cc:1428] Setting key 7291006517480b46221f7b557192d57d
I20260811 04:40:43.223019 25659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:43.229106 25659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135: Bootstrap starting.
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 04:40:43.231544 25659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:43.232385 25659 log.cc:824] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:43.234270 25659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135: No bootstrap required, opened a new log
I20260811 04:40:43.236326 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:43.223907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:33609 (local address 127.0.0.1:42149)
0811 04:40:43.224333 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:43.224348 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:43.224383 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:40:43.225157 (+ 774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:43.225176 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:43.225512 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:43.225874 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:43.226745 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:43.227702 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:43.228549 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:43.229011 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:43.229174 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:43.231742 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:43.231747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:43.231761 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:43.231788 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:43.233389 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:43.233919 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:43.233924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:43.233928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:43.234050 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:43.234419 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:43.234422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:43.234423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:43.234890 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:43.235103 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:43.235452 (+ 349us) server_negotiation.cc:299] Negotiation successful
0811 04:40:43.235665 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":134,"threads_started":1}
I20260811 04:40:43.237852 25659 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } }
I20260811 04:40:43.238097 25659 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:43.238214 25659 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbb49a2ce38643c0a9203880e2f3c135, State: Initialized, Role: FOLLOWER
I20260811 04:40:43.238698 25659 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } }
I20260811 04:40:43.238822 25659 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:43.238906 25659 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:43.239004 25659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:43.239173 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:43.239888 25659 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } }
I20260811 04:40:43.240217 25659 leader_election.cc:302] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbb49a2ce38643c0a9203880e2f3c135; no voters:
I20260811 04:40:43.240587 25659 leader_election.cc:288] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:43.240691 25668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:43.241536 25668 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [term 1 LEADER]: Becoming Leader. State: Replica: dbb49a2ce38643c0a9203880e2f3c135, State: Running, Role: LEADER
I20260811 04:40:43.241899 25659 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:43.241952 25668 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } }
I20260811 04:40:43.243587 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbb49a2ce38643c0a9203880e2f3c135. Latest consensus state: current_term: 1 leader_uuid: "dbb49a2ce38643c0a9203880e2f3c135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } } }
I20260811 04:40:43.243719 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:43.244006 25669 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbb49a2ce38643c0a9203880e2f3c135" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbb49a2ce38643c0a9203880e2f3c135" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42149 } } }
I20260811 04:40:43.244334 25669 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:43.244287 25686 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:43.247624 25686 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:43.250283 25662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:43.241195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:56315 (local address 127.0.0.1:42149)
0811 04:40:43.241430 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:43.241436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:43.241464 (+ 28us) server_negotiation.cc:407] Connection header received
0811 04:40:43.241611 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:43.241614 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:43.241662 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:43.241713 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:43.242626 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:43.243354 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:43.244532 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:43.244839 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:43.244910 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:43.246596 (+ 1686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:43.246599 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:43.246602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:43.246627 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:43.248251 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:43.248814 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:43.248818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:43.248820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:43.248905 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:43.249300 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:43.249304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:43.249306 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:43.249531 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:43.249684 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:43.250028 (+ 344us) server_negotiation.cc:299] Negotiation successful
0811 04:40:43.250125 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160}
I20260811 04:40:43.253192 25686 catalog_manager.cc:1347] Generated new cluster ID: 2d45a1673db64d07b33d2dd3a87a13a2
I20260811 04:40:43.253258 25686 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:43.289928 25686 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:43.291252 25686 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:43.304440 25686 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbb49a2ce38643c0a9203880e2f3c135: Generated new TSK 0
I20260811 04:40:43.305308 25686 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:43.309177 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25586
W20260811 04:40:43.317348 25568 connection.cc:535] client connection to 127.0.0.1:42149 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42149): Connection reset by peer (error 104)
2026-08-11T04:40:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25693](info): set up 2 sockets
Aug 11 04:40:43 dist-test-slave-0kls krb5kdc[25693](info): commencing operation
krb5kdc: starting...
W20260811 04:40:44.370332 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:44 dist-test-slave-0kls krb5kdc[25693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:44.497311 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36223
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:44.606549 25707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:44.606858 25707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:44.606932 25707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:44.611227 25707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:44.611335 25707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:44.611382 25707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:44.611438 25707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:44.611474 25707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:44.616631 25707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:44.617690 25707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:44.618781 25707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260811 04:40:44.627223 25707 server_base.cc:1034] running on GCE node
W20260811 04:40:44.627381 25714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:44.627364 25716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:44.627363 25713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:44.628127 25707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:44.629199 25707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:44.630388 25707 hybrid_clock.cc:648] HybridClock initialized: now 1786423244630373 us; error 33 us; skew 500 ppm
Aug 11 04:40:44 dist-test-slave-0kls krb5kdc[25693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:44.633193 25707 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:44.634078 25707 webserver.cc:460] Webserver started at http://127.0.0.1:39909/ using document root <none> and password file <none>
I20260811 04:40:44.634614 25707 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:44.634706 25707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:44.634929 25707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:44.636925 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "40de7744cc4840bd9469e5a66f37cc48"
format_stamp: "Formatted at 2026-08-11 04:40:44 on dist-test-slave-0kls"
server_key: "7e9a0e231ec41e4a52c065313ae4d613"
server_key_iv: "b9e7d2d9d88630b2fd0fe5870a0f20ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:44.637557 25707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "40de7744cc4840bd9469e5a66f37cc48"
format_stamp: "Formatted at 2026-08-11 04:40:44 on dist-test-slave-0kls"
server_key: "7e9a0e231ec41e4a52c065313ae4d613"
server_key_iv: "b9e7d2d9d88630b2fd0fe5870a0f20ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:44.641295 25707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260811 04:40:44.643828 25725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:44.645027 25707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260811 04:40:44.645193 25707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "40de7744cc4840bd9469e5a66f37cc48"
format_stamp: "Formatted at 2026-08-11 04:40:44 on dist-test-slave-0kls"
server_key: "7e9a0e231ec41e4a52c065313ae4d613"
server_key_iv: "b9e7d2d9d88630b2fd0fe5870a0f20ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:44.645306 25707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:44.655617 25707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:44.658346 25707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:44.658540 25707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:44.665678 25707 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36223
I20260811 04:40:44.665705 25779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36223 every 8 connection(s)
I20260811 04:40:44.666615 25707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:44.669390 25780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:44.674226 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25707
I20260811 04:40:44.674369 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:44.674664 3002 external_mini_cluster.cc:1428] Setting key 54b0240934ee346078ea4f1b10cefc39
I20260811 04:40:44.674998 25780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48: Bootstrap starting.
I20260811 04:40:44.677513 25780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:44.678279 25780 log.cc:824] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:44.680054 25780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48: No bootstrap required, opened a new log
I20260811 04:40:44.683398 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:44.676068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:58501 (local address 127.0.0.1:36223)
0811 04:40:44.676511 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:44.676527 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:44.676572 (+ 45us) server_negotiation.cc:407] Connection header received
0811 04:40:44.677159 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:44.677173 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:44.677505 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:44.677886 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:44.678298 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:44.679286 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:44.680182 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:44.680662 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:44.680890 (+ 228us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:44.681828 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:44.681836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:44.681851 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:44.682226 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:44.682278 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:44.682300 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:44.682439 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:44.682627 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 04:40:44.682757 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":125,"threads_started":1}
I20260811 04:40:44.683796 25780 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260811 04:40:44.684020 25780 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:44.684104 25780 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40de7744cc4840bd9469e5a66f37cc48, State: Initialized, Role: FOLLOWER
I20260811 04:40:44.684573 25780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260811 04:40:44.684725 25780 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:44.684787 25780 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:44.684873 25780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:44.685971 25780 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260811 04:40:44.686324 25780 leader_election.cc:302] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40de7744cc4840bd9469e5a66f37cc48; no voters:
I20260811 04:40:44.686400 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:44.686703 25780 leader_election.cc:288] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:44.686760 25786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:44.686965 25786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [term 1 LEADER]: Becoming Leader. State: Replica: 40de7744cc4840bd9469e5a66f37cc48, State: Running, Role: LEADER
I20260811 04:40:44.687288 25786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } }
I20260811 04:40:44.687891 25780 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:44.688386 25786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40de7744cc4840bd9469e5a66f37cc48. Latest consensus state: current_term: 1 leader_uuid: "40de7744cc4840bd9469e5a66f37cc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } } }
I20260811 04:40:44.688556 25786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:44.689040 25799 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:44.689316 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40de7744cc4840bd9469e5a66f37cc48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40de7744cc4840bd9469e5a66f37cc48" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36223 } } }
I20260811 04:40:44.689404 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:44.692193 25799 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:44.693534 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:44.688552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:38965 (local address 127.0.0.1:36223)
0811 04:40:44.688660 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:44.688664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:44.689117 (+ 453us) server_negotiation.cc:407] Connection header received
0811 04:40:44.689355 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:44.689358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:44.689395 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:44.689437 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:44.689879 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:44.690595 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:44.691660 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:44.691960 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:44.692031 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:44.693037 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:44.693041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:44.693043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:44.693056 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:44.693084 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:44.693097 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:44.693192 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:44.693380 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 04:40:44.693414 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260811 04:40:44.697471 25799 catalog_manager.cc:1347] Generated new cluster ID: 175c145003bc409bbcc648f6157cafbd
I20260811 04:40:44.697530 25799 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:44.703390 25799 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:44.704203 25799 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:44.710475 25799 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40de7744cc4840bd9469e5a66f37cc48: Generated new TSK 0
I20260811 04:40:44.711061 25799 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:44.715907 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25707
W20260811 04:40:44.723670 25691 connection.cc:535] client connection to 127.0.0.1:36223 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36223): Connection reset by peer (error 104)
2026-08-11T04:40:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:44 dist-test-slave-0kls krb5kdc[25814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:44 dist-test-slave-0kls krb5kdc[25814](info): set up 2 sockets
Aug 11 04:40:44 dist-test-slave-0kls krb5kdc[25814](info): commencing operation
krb5kdc: starting...
W20260811 04:40:45.767356 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:45 dist-test-slave-0kls krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:45.898233 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:46.017436 25828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:46.017827 25828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:46.017904 25828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:46.024878 25828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:46.024955 25828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:46.024989 25828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:46.025023 25828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:46.025053 25828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:46.034111 25828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:46.035295 25828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:46.036369 25828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:46.045046 25834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:46.045068 25837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:46.045058 25835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:46.045403 25828 server_base.cc:1034] running on GCE node
I20260811 04:40:46.046064 25828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:46.047153 25828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:46.048326 25828 hybrid_clock.cc:648] HybridClock initialized: now 1786423246048307 us; error 30 us; skew 500 ppm
Aug 11 04:40:46 dist-test-slave-0kls krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:46.051095 25828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:46.051880 25828 webserver.cc:460] Webserver started at http://127.0.0.1:36429/ using document root <none> and password file <none>
I20260811 04:40:46.052393 25828 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:46.052445 25828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:46.052603 25828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:46.054615 25828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3"
format_stamp: "Formatted at 2026-08-11 04:40:46 on dist-test-slave-0kls"
server_key: "465f27937274e15fa580f82c384b8216"
server_key_iv: "6dc7f84b1d7730f76a90e9e84d9b7db3"
server_key_version: "encryptionkey@0"
I20260811 04:40:46.055146 25828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3"
format_stamp: "Formatted at 2026-08-11 04:40:46 on dist-test-slave-0kls"
server_key: "465f27937274e15fa580f82c384b8216"
server_key_iv: "6dc7f84b1d7730f76a90e9e84d9b7db3"
server_key_version: "encryptionkey@0"
I20260811 04:40:46.058811 25828 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:40:46.061313 25845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:46.062559 25828 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:40:46.062702 25828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3"
format_stamp: "Formatted at 2026-08-11 04:40:46 on dist-test-slave-0kls"
server_key: "465f27937274e15fa580f82c384b8216"
server_key_iv: "6dc7f84b1d7730f76a90e9e84d9b7db3"
server_key_version: "encryptionkey@0"
I20260811 04:40:46.062812 25828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:46.067200 25828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:46.069990 25828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:46.070174 25828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:46.077183 25828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41213
I20260811 04:40:46.077206 25902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41213 every 8 connection(s)
I20260811 04:40:46.078073 25828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:46.080704 25903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:46.085660 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25828
I20260811 04:40:46.085829 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:46.085932 25903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3: Bootstrap starting.
I20260811 04:40:46.086156 3002 external_mini_cluster.cc:1428] Setting key 6c750db9585ecb758faad2061261a83c
I20260811 04:40:46.088423 25903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:46.089149 25903 log.cc:824] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:46.091161 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:46.087447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:56413 (local address 127.0.0.1:41213)
0811 04:40:46.087860 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:46.087876 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:46.087909 (+ 33us) server_negotiation.cc:407] Connection header received
0811 04:40:46.088544 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:46.088568 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:46.088860 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:46.089179 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:46.089993 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:46.089998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:46.090014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:46.090028 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:46.090089 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:46.090110 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:46.090202 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:46.090364 (+ 162us) server_negotiation.cc:299] Negotiation successful
0811 04:40:46.090525 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":122,"threads_started":1}
I20260811 04:40:46.091262 25903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3: No bootstrap required, opened a new log
I20260811 04:40:46.093720 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:46.095508 25903 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260811 04:40:46.095817 25903 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:46.095890 25903 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02c8c8f5c4747b2ae5e9fc37db38fd3, State: Initialized, Role: FOLLOWER
I20260811 04:40:46.096364 25903 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260811 04:40:46.096537 25903 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:46.096660 25903 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:46.096771 25903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:46.097669 25906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:46.096031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:50863 (local address 127.0.0.1:41213)
0811 04:40:46.096178 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:46.096184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:46.096421 (+ 237us) server_negotiation.cc:407] Connection header received
0811 04:40:46.096586 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:46.096589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:46.096624 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:46.096712 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:46.097297 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:46.097300 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:46.097301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:46.097312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:46.097334 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:46.097343 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:46.097416 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:46.097511 (+ 95us) server_negotiation.cc:299] Negotiation successful
0811 04:40:46.097539 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:40:46.098048 25903 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260811 04:40:46.098410 25903 leader_election.cc:302] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02c8c8f5c4747b2ae5e9fc37db38fd3; no voters:
I20260811 04:40:46.098747 25903 leader_election.cc:288] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:46.098865 25913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:46.099071 25913 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [term 1 LEADER]: Becoming Leader. State: Replica: d02c8c8f5c4747b2ae5e9fc37db38fd3, State: Running, Role: LEADER
I20260811 04:40:46.099370 25913 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } }
I20260811 04:40:46.099960 25903 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:46.101038 25915 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02c8c8f5c4747b2ae5e9fc37db38fd3. Latest consensus state: current_term: 1 leader_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260811 04:40:46.101199 25915 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:46.101466 25914 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c8c8f5c4747b2ae5e9fc37db38fd3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41213 } } }
I20260811 04:40:46.101562 25914 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:46.101692 25922 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:46.104727 25922 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:46.109896 25922 catalog_manager.cc:1347] Generated new cluster ID: 0afc4d45d92b4119b56b31ace1cbb871
I20260811 04:40:46.109956 25922 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:46.143083 25922 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:46.143435 25922 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:46.150965 25922 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02c8c8f5c4747b2ae5e9fc37db38fd3: Generated new TSK 0
I20260811 04:40:46.151754 25922 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:46.165113 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25828
2026-08-11T04:40:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:46 dist-test-slave-0kls krb5kdc[25935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:46 dist-test-slave-0kls krb5kdc[25935](info): set up 2 sockets
Aug 11 04:40:46 dist-test-slave-0kls krb5kdc[25935](info): commencing operation
krb5kdc: starting...
W20260811 04:40:47.224987 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:47.352113 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:47.463320 25951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:47.463605 25951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:47.463685 25951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:47.467484 25951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:47.467574 25951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:47.467612 25951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:47.467655 25951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:47.467687 25951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:47.472827 25951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:47.474018 25951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:47.475037 25951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:47.483922 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:47.483940 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:47.483986 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:47.484232 25951 server_base.cc:1034] running on GCE node
I20260811 04:40:47.484896 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:47.486153 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:47.487346 25951 hybrid_clock.cc:648] HybridClock initialized: now 1786423247487310 us; error 31 us; skew 500 ppm
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[25935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:47.490238 25951 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:47.491137 25951 webserver.cc:460] Webserver started at http://127.0.0.1:43963/ using document root <none> and password file <none>
I20260811 04:40:47.491734 25951 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:47.491828 25951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:47.492074 25951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:47.494251 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "0a00e906f9c042f6b24603f928d8215e"
format_stamp: "Formatted at 2026-08-11 04:40:47 on dist-test-slave-0kls"
server_key: "4e2a955b708a6025ac828e6c568a68bd"
server_key_iv: "bf1febd97baa270f842d6e116c887ec5"
server_key_version: "encryptionkey@0"
I20260811 04:40:47.494827 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "0a00e906f9c042f6b24603f928d8215e"
format_stamp: "Formatted at 2026-08-11 04:40:47 on dist-test-slave-0kls"
server_key: "4e2a955b708a6025ac828e6c568a68bd"
server_key_iv: "bf1febd97baa270f842d6e116c887ec5"
server_key_version: "encryptionkey@0"
I20260811 04:40:47.498764 25951 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260811 04:40:47.501206 25966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:47.502372 25951 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:47.502504 25951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "0a00e906f9c042f6b24603f928d8215e"
format_stamp: "Formatted at 2026-08-11 04:40:47 on dist-test-slave-0kls"
server_key: "4e2a955b708a6025ac828e6c568a68bd"
server_key_iv: "bf1febd97baa270f842d6e116c887ec5"
server_key_version: "encryptionkey@0"
I20260811 04:40:47.502617 25951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:47.521411 25951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:47.525012 25951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:47.525239 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:47.532895 25951 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39985
I20260811 04:40:47.532904 26023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39985 every 8 connection(s)
I20260811 04:40:47.533874 25951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:47.536657 26024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:47.538326 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 25951
I20260811 04:40:47.538484 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:47.538806 3002 external_mini_cluster.cc:1428] Setting key 6400bf715aa04a0f86a8a4467ca04297
I20260811 04:40:47.543110 26024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e: Bootstrap starting.
I20260811 04:40:47.545624 26024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:47.546599 26024 log.cc:824] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[25935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260811 04:40:47.548753 26024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e: No bootstrap required, opened a new log
I20260811 04:40:47.553819 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:47.540191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:51981 (local address 127.0.0.1:39985)
0811 04:40:47.540605 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:47.540622 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:47.540660 (+ 38us) server_negotiation.cc:407] Connection header received
0811 04:40:47.541327 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:47.541343 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:47.541664 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:47.542006 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:47.542882 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:47.543889 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:47.545047 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:47.545524 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:47.545654 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:47.548402 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:47.548412 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:47.548429 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:47.548461 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:47.550516 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:47.551067 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:47.551073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:47.551095 (+ 22us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:47.551265 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:47.551609 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:47.551614 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:47.551617 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:47.552145 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:47.552409 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:47.552607 (+ 198us) server_negotiation.cc:299] Negotiation successful
0811 04:40:47.552830 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":127,"threads_started":1}
I20260811 04:40:47.553748 26024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260811 04:40:47.554067 26024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:47.554152 26024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a00e906f9c042f6b24603f928d8215e, State: Initialized, Role: FOLLOWER
I20260811 04:40:47.554661 26024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260811 04:40:47.554858 26024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:47.554934 26024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:47.555054 26024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:47.556180 26024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260811 04:40:47.556581 26024 leader_election.cc:302] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a00e906f9c042f6b24603f928d8215e; no voters:
I20260811 04:40:47.556947 26024 leader_election.cc:288] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:47.557041 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:47.557034 26029 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:47.557271 26029 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [term 1 LEADER]: Becoming Leader. State: Replica: 0a00e906f9c042f6b24603f928d8215e, State: Running, Role: LEADER
I20260811 04:40:47.557561 26029 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } }
I20260811 04:40:47.558501 26024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:47.559747 26030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a00e906f9c042f6b24603f928d8215e. Latest consensus state: current_term: 1 leader_uuid: "0a00e906f9c042f6b24603f928d8215e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260811 04:40:47.559906 26030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:47.560016 26031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a00e906f9c042f6b24603f928d8215e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a00e906f9c042f6b24603f928d8215e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39985 } } }
I20260811 04:40:47.560161 26031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:47.560341 26039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:47.565135 26039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:47.569208 26027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:47.559843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:42937 (local address 127.0.0.1:39985)
0811 04:40:47.560220 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:47.560224 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:47.560238 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:40:47.560274 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:47.560277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:47.560323 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:47.560409 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:47.561350 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:47.562203 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:47.563244 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:47.563564 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:47.563687 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:47.565299 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:47.565303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:47.565306 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:47.565334 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:47.567160 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:47.567717 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:47.567721 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:47.567723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:47.567816 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:47.568214 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:47.568218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:47.568220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:47.568459 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:47.568609 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:47.568934 (+ 325us) server_negotiation.cc:299] Negotiation successful
0811 04:40:47.569048 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283}
I20260811 04:40:47.570942 26039 catalog_manager.cc:1347] Generated new cluster ID: ddc845d2fa0f49158374e630668752a9
I20260811 04:40:47.571017 26039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:47.588634 26039 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:47.589823 26039 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:47.596900 26039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a00e906f9c042f6b24603f928d8215e: Generated new TSK 0
I20260811 04:40:47.597641 26039 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:47.607649 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 25951
W20260811 04:40:47.616813 25933 connection.cc:535] client connection to 127.0.0.1:39985 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39985): Connection reset by peer (error 104)
2026-08-11T04:40:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1443 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[26060](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[26060](info): set up 2 sockets
Aug 11 04:40:47 dist-test-slave-0kls krb5kdc[26060](info): commencing operation
krb5kdc: starting...
W20260811 04:40:48.684396 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:48 dist-test-slave-0kls krb5kdc[26060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:48.816771 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:48.928979 26073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:48.929260 26073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:48.929299 26073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:48.933449 26073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:48.933501 26073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:48.933521 26073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:48.933537 26073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:48.933552 26073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:48.938663 26073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33343
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:48.939577 26073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:48.940529 26073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:48.948876 26083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:48.948887 26081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:48.948833 26080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:48.949173 26073 server_base.cc:1034] running on GCE node
I20260811 04:40:48.949750 26073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:48.950760 26073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:48.951941 26073 hybrid_clock.cc:648] HybridClock initialized: now 1786423248951911 us; error 41 us; skew 500 ppm
Aug 11 04:40:48 dist-test-slave-0kls krb5kdc[26060](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423248, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:48.954707 26073 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:48.955586 26073 webserver.cc:460] Webserver started at http://127.0.0.1:46209/ using document root <none> and password file <none>
I20260811 04:40:48.956151 26073 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:48.956243 26073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:48.956475 26073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:48.958506 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "dbcf681d772a4952b1f311ebe4740c8e"
format_stamp: "Formatted at 2026-08-11 04:40:48 on dist-test-slave-0kls"
server_key: "1d77e8577427487cf19954485190788f"
server_key_iv: "a065349e9e4e10f07c6f463a82c864d0"
server_key_version: "encryptionkey@0"
I20260811 04:40:48.959093 26073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "dbcf681d772a4952b1f311ebe4740c8e"
format_stamp: "Formatted at 2026-08-11 04:40:48 on dist-test-slave-0kls"
server_key: "1d77e8577427487cf19954485190788f"
server_key_iv: "a065349e9e4e10f07c6f463a82c864d0"
server_key_version: "encryptionkey@0"
I20260811 04:40:48.962816 26073 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260811 04:40:48.965165 26090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:48.966498 26073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260811 04:40:48.966646 26073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "dbcf681d772a4952b1f311ebe4740c8e"
format_stamp: "Formatted at 2026-08-11 04:40:48 on dist-test-slave-0kls"
server_key: "1d77e8577427487cf19954485190788f"
server_key_iv: "a065349e9e4e10f07c6f463a82c864d0"
server_key_version: "encryptionkey@0"
I20260811 04:40:48.966759 26073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:48.990585 26073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:48.993211 26073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:48.993430 26073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:49.000072 26073 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33343
I20260811 04:40:49.000109 26147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33343 every 8 connection(s)
I20260811 04:40:49.000973 26073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:49.003821 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26073
I20260811 04:40:49.003986 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:49.003984 26149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:49.004371 3002 external_mini_cluster.cc:1428] Setting key 375dc27d5e0d6256dbb37e627bba52a5
I20260811 04:40:49.011466 26149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: Bootstrap starting.
I20260811 04:40:49.013489 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:49.005686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:44541 (local address 127.0.0.1:33343)
0811 04:40:49.006149 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:49.006167 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:49.006213 (+ 46us) server_negotiation.cc:407] Connection header received
0811 04:40:49.006836 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:49.006852 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:49.007167 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:49.007429 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:49.007874 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:49.008848 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:49.010186 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:49.010639 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:49.010763 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:49.011742 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:49.011747 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:49.011760 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:49.012123 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:49.012190 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:49.012213 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:49.012317 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:49.012546 (+ 229us) server_negotiation.cc:299] Negotiation successful
0811 04:40:49.012692 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":155,"threads_started":1}
I20260811 04:40:49.014037 26149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:49.014787 26149 log.cc:824] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:49.016494 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:49.017060 26149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: No bootstrap required, opened a new log
I20260811 04:40:49.021304 26149 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260811 04:40:49.021616 26149 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:49.021700 26149 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcf681d772a4952b1f311ebe4740c8e, State: Initialized, Role: FOLLOWER
I20260811 04:40:49.022207 26149 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260811 04:40:49.022383 26149 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:49.022459 26149 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:49.022567 26149 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:49.022964 26152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:49.018586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:34111 (local address 127.0.0.1:33343)
0811 04:40:49.018696 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:49.018701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:49.018889 (+ 188us) server_negotiation.cc:407] Connection header received
0811 04:40:49.018964 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:49.018968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:49.019018 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:49.019113 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:49.019383 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:49.020140 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:49.020972 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:49.021366 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:49.021476 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:49.022414 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:49.022418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:49.022422 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:49.022438 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:49.022468 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:49.022485 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:49.022583 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:49.022783 (+ 200us) server_negotiation.cc:299] Negotiation successful
0811 04:40:49.022821 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260811 04:40:49.023785 26149 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260811 04:40:49.024119 26149 leader_election.cc:302] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcf681d772a4952b1f311ebe4740c8e; no voters:
I20260811 04:40:49.024494 26149 leader_election.cc:288] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:49.024590 26159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:49.024839 26159 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [term 1 LEADER]: Becoming Leader. State: Replica: dbcf681d772a4952b1f311ebe4740c8e, State: Running, Role: LEADER
I20260811 04:40:49.025242 26159 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } }
I20260811 04:40:49.025826 26149 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:49.026912 26160 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcf681d772a4952b1f311ebe4740c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260811 04:40:49.027048 26160 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:49.027385 26161 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcf681d772a4952b1f311ebe4740c8e. Latest consensus state: current_term: 1 leader_uuid: "dbcf681d772a4952b1f311ebe4740c8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcf681d772a4952b1f311ebe4740c8e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33343 } } }
I20260811 04:40:49.028213 26161 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:49.030611 26177 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:49.030680 26177 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:49.030743 26168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:49.031412 26168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:49.036166 26168 catalog_manager.cc:1347] Generated new cluster ID: eb11c03493084b9f950a847cd9ff3be9
I20260811 04:40:49.036227 26168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:49.048941 26168 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:49.050197 26168 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:49.055269 26168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcf681d772a4952b1f311ebe4740c8e: Generated new TSK 0
I20260811 04:40:49.055888 26168 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:49.066207 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26073
W20260811 04:40:49.075289 26058 connection.cc:535] client connection to 127.0.0.1:33343 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33343): Connection reset by peer (error 104)
2026-08-11T04:40:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1458 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:49 dist-test-slave-0kls krb5kdc[26185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:49 dist-test-slave-0kls krb5kdc[26185](info): set up 2 sockets
Aug 11 04:40:49 dist-test-slave-0kls krb5kdc[26185](info): commencing operation
krb5kdc: starting...
W20260811 04:40:50.139220 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:50 dist-test-slave-0kls krb5kdc[26185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260811 04:40:50.270943 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:50.386626 26200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:50.386919 26200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:50.386960 26200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:50.391000 26200 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:50.391073 26200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:50.391091 26200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:50.391108 26200 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:50.391121 26200 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:50.396337 26200 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40491
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:50.397411 26200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:50.398865 26200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:50.407665 26207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:50.407676 26209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:50.407720 26205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:50.407814 26200 server_base.cc:1034] running on GCE node
I20260811 04:40:50.408501 26200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:50.409641 26200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:50.410815 26200 hybrid_clock.cc:648] HybridClock initialized: now 1786423250410777 us; error 50 us; skew 500 ppm
Aug 11 04:40:50 dist-test-slave-0kls krb5kdc[26185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:50.413566 26200 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260811 04:40:50.414469 26200 webserver.cc:460] Webserver started at http://127.0.0.1:37329/ using document root <none> and password file <none>
I20260811 04:40:50.415011 26200 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:50.415077 26200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:50.415243 26200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:50.417335 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "b4d08b61532548168b07b96846c6e07b"
format_stamp: "Formatted at 2026-08-11 04:40:50 on dist-test-slave-0kls"
server_key: "41ca5e1fe88fef85b45ae85f951a364c"
server_key_iv: "040d43c6711d31e385cff2b300d0b29c"
server_key_version: "encryptionkey@0"
I20260811 04:40:50.418012 26200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "b4d08b61532548168b07b96846c6e07b"
format_stamp: "Formatted at 2026-08-11 04:40:50 on dist-test-slave-0kls"
server_key: "41ca5e1fe88fef85b45ae85f951a364c"
server_key_iv: "040d43c6711d31e385cff2b300d0b29c"
server_key_version: "encryptionkey@0"
I20260811 04:40:50.422029 26200 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:40:50.424546 26217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:50.425946 26200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:50.426077 26200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "b4d08b61532548168b07b96846c6e07b"
format_stamp: "Formatted at 2026-08-11 04:40:50 on dist-test-slave-0kls"
server_key: "41ca5e1fe88fef85b45ae85f951a364c"
server_key_iv: "040d43c6711d31e385cff2b300d0b29c"
server_key_version: "encryptionkey@0"
I20260811 04:40:50.426203 26200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:50.431107 26200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:50.434069 26200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:50.434274 26200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:50.441273 26200 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40491
I20260811 04:40:50.441332 26275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40491 every 8 connection(s)
I20260811 04:40:50.442283 26200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:50.445176 26276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:50.448251 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26200
I20260811 04:40:50.448426 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:50.448735 3002 external_mini_cluster.cc:1428] Setting key 6be07435c2a5c5af9e70c275bf301c66
I20260811 04:40:50.451416 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b: Bootstrap starting.
I20260811 04:40:50.454195 26276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:50.454545 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:50.450199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:38551 (local address 127.0.0.1:40491)
0811 04:40:50.450634 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:50.450660 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:50.450708 (+ 48us) server_negotiation.cc:407] Connection header received
0811 04:40:50.451534 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:50.451557 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:50.451916 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:50.452284 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:50.453092 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:50.453097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:50.453111 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:50.453124 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:50.453197 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:50.453223 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:50.453355 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:50.453420 (+ 65us) server_negotiation.cc:299] Negotiation successful
0811 04:40:50.453574 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":113,"threads_started":1}
I20260811 04:40:50.455075 26276 log.cc:824] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:50.457127 26276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b: No bootstrap required, opened a new log
I20260811 04:40:50.457406 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:50.461634 26276 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260811 04:40:50.461967 26276 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:50.462044 26276 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d08b61532548168b07b96846c6e07b, State: Initialized, Role: FOLLOWER
I20260811 04:40:50.462575 26276 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260811 04:40:50.462767 26276 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:50.462857 26276 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:50.462958 26276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:50.463727 26281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:50.459813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.104:47127 (local address 127.0.0.1:40491)
0811 04:40:50.459945 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:50.459949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:50.462195 (+ 2246us) server_negotiation.cc:407] Connection header received
0811 04:40:50.462285 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:50.462288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:50.462331 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:50.462421 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:50.463313 (+ 892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:50.463316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:50.463318 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:50.463329 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:50.463355 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:50.463366 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:50.463438 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:50.463544 (+ 106us) server_negotiation.cc:299] Negotiation successful
0811 04:40:50.463577 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:40:50.464179 26276 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260811 04:40:50.464550 26276 leader_election.cc:302] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d08b61532548168b07b96846c6e07b; no voters:
I20260811 04:40:50.464896 26276 leader_election.cc:288] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:50.465029 26297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:50.465299 26297 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [term 1 LEADER]: Becoming Leader. State: Replica: b4d08b61532548168b07b96846c6e07b, State: Running, Role: LEADER
I20260811 04:40:50.465678 26297 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } }
I20260811 04:40:50.466315 26276 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:50.467552 26299 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d08b61532548168b07b96846c6e07b. Latest consensus state: current_term: 1 leader_uuid: "b4d08b61532548168b07b96846c6e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260811 04:40:50.467545 26298 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d08b61532548168b07b96846c6e07b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d08b61532548168b07b96846c6e07b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40491 } } }
I20260811 04:40:50.467703 26299 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:50.467703 26298 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:50.468130 26306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:50.471184 26306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:50.477247 26306 catalog_manager.cc:1347] Generated new cluster ID: 3eeb481bf74d4ffab41dadc6ea77fed6
I20260811 04:40:50.477322 26306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:50.515306 26306 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:50.515710 26306 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:50.542052 26306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d08b61532548168b07b96846c6e07b: Generated new TSK 0
I20260811 04:40:50.543035 26306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:50.548213 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26200
2026-08-11T04:40:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1480 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:50 dist-test-slave-0kls krb5kdc[26323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:50 dist-test-slave-0kls krb5kdc[26323](info): set up 2 sockets
Aug 11 04:40:50 dist-test-slave-0kls krb5kdc[26323](info): commencing operation
krb5kdc: starting...
W20260811 04:40:51.638373 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:51 dist-test-slave-0kls krb5kdc[26323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:51.772636 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36707
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:36707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:51.888104 26338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:51.888517 26338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:51.888602 26338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:51.892866 26338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:51.892931 26338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:51.892952 26338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:51.892969 26338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:51.893014 26338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:51.898958 26338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:36707
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:51.900115 26338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:51.901194 26338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:51.910156 26344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:51.910172 26348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:51.910151 26345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:51.910351 26338 server_base.cc:1034] running on GCE node
I20260811 04:40:51.911113 26338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:51.912189 26338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:51.913393 26338 hybrid_clock.cc:648] HybridClock initialized: now 1786423251913351 us; error 32 us; skew 500 ppm
Aug 11 04:40:51 dist-test-slave-0kls krb5kdc[26323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423251, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:51.920396 26338 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:51.921411 26338 webserver.cc:460] Webserver started at http://127.2.238.190:42559/ using document root <none> and password file <none>
I20260811 04:40:51.922111 26338 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:51.922214 26338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:51.922472 26338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:51.924639 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "c59569cfd2a54d31ac31aa74ea08d1f6"
format_stamp: "Formatted at 2026-08-11 04:40:51 on dist-test-slave-0kls"
server_key: "e034678fbbcadc8afa74035992245507"
server_key_iv: "717041b2f17ecaccaa87595b3b57ed53"
server_key_version: "encryptionkey@0"
I20260811 04:40:51.925283 26338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "c59569cfd2a54d31ac31aa74ea08d1f6"
format_stamp: "Formatted at 2026-08-11 04:40:51 on dist-test-slave-0kls"
server_key: "e034678fbbcadc8afa74035992245507"
server_key_iv: "717041b2f17ecaccaa87595b3b57ed53"
server_key_version: "encryptionkey@0"
I20260811 04:40:51.929167 26338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260811 04:40:51.931762 26356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:51.932888 26338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:51.933061 26338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "c59569cfd2a54d31ac31aa74ea08d1f6"
format_stamp: "Formatted at 2026-08-11 04:40:51 on dist-test-slave-0kls"
server_key: "e034678fbbcadc8afa74035992245507"
server_key_iv: "717041b2f17ecaccaa87595b3b57ed53"
server_key_version: "encryptionkey@0"
I20260811 04:40:51.933177 26338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:51.958384 26338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:51.961326 26338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:51.961520 26338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:51.968639 26338 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:36707
I20260811 04:40:51.968675 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:36707 every 8 connection(s)
I20260811 04:40:51.969558 26338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:51.970383 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26338
I20260811 04:40:51.970515 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:51.970852 3002 external_mini_cluster.cc:1428] Setting key ca1e4da591e0f6a0d05e2973b80e7f2d
I20260811 04:40:51.972909 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:51.978689 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: Bootstrap starting.
Aug 11 04:40:51 dist-test-slave-0kls krb5kdc[26323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:51.981287 26415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:51.982151 26415 log.cc:824] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:51.984380 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: No bootstrap required, opened a new log
I20260811 04:40:51.985539 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:51.972324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54031 (local address 127.2.238.190:36707)
0811 04:40:51.972781 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:51.972796 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:51.972831 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:40:51.973517 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:51.973534 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:51.973950 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:51.974304 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:51.975149 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:51.976139 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:51.976999 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:51.977486 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:51.977676 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:51.980500 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:51.980505 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:51.980517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:51.980545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:51.982338 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:51.982954 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:51.982961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:51.982970 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:51.983102 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:51.983445 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:51.983448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:51.983450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:51.983929 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:51.984159 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:51.984365 (+ 206us) server_negotiation.cc:299] Negotiation successful
0811 04:40:51.984609 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":149,"threads_started":1}
I20260811 04:40:51.988266 26415 raft_consensus.cc:348] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } }
I20260811 04:40:51.988601 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:51.988655 26415 raft_consensus.cc:374] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:51.988711 26415 raft_consensus.cc:729] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59569cfd2a54d31ac31aa74ea08d1f6, State: Initialized, Role: FOLLOWER
I20260811 04:40:51.989176 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } }
I20260811 04:40:51.989341 26415 raft_consensus.cc:388] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:51.989414 26415 raft_consensus.cc:482] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:51.989562 26415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:51.990783 26415 raft_consensus.cc:504] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } }
I20260811 04:40:51.991147 26415 leader_election.cc:302] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c59569cfd2a54d31ac31aa74ea08d1f6; no voters:
I20260811 04:40:51.991472 26415 leader_election.cc:288] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:51.991575 26425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:51.991822 26425 raft_consensus.cc:686] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [term 1 LEADER]: Becoming Leader. State: Replica: c59569cfd2a54d31ac31aa74ea08d1f6, State: Running, Role: LEADER
I20260811 04:40:51.992225 26425 consensus_queue.cc:237] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } }
I20260811 04:40:51.992710 26415 sys_catalog.cc:564] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:51.993456 26425 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c59569cfd2a54d31ac31aa74ea08d1f6. Latest consensus state: current_term: 1 leader_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } } }
I20260811 04:40:51.994236 26425 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:51.994136 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c59569cfd2a54d31ac31aa74ea08d1f6" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 36707 } } }
I20260811 04:40:51.994601 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6 [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:51.998901 26444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:51.998977 26444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:51.999039 26441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:51.999979 26441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:52.000087 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:51.990881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42655 (local address 127.2.238.190:36707)
0811 04:40:51.990984 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:51.990988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:51.991162 (+ 174us) server_negotiation.cc:407] Connection header received
0811 04:40:51.991241 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:51.991245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:51.991292 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:51.991363 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:51.992178 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:51.992930 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:51.994024 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:51.994379 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:51.994476 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:51.996211 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:51.996214 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:51.996217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:51.996240 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:51.997740 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:51.998289 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:51.998293 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:51.998294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:51.998378 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:51.998759 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:51.998762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:51.998764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:51.998978 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:51.999124 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:51.999772 (+ 648us) server_negotiation.cc:299] Negotiation successful
0811 04:40:51.999900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260811 04:40:52.006417 26441 catalog_manager.cc:1347] Generated new cluster ID: 3e62479e6cc2443e8247379324f31086
I20260811 04:40:52.006479 26441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:52.029994 26441 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:52.030896 26441 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:52.038460 26441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c59569cfd2a54d31ac31aa74ea08d1f6: Generated new TSK 0
I20260811 04:40:52.039179 26441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:52.050246 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26338
W20260811 04:40:52.059834 26320 connection.cc:535] client connection to 127.2.238.190:36707 recv error: Network error: failed to read from TLS socket (remote: 127.2.238.190:36707): Connection reset by peer (error 104)
2026-08-11T04:40:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:52 dist-test-slave-0kls krb5kdc[26450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:52 dist-test-slave-0kls krb5kdc[26450](info): set up 2 sockets
Aug 11 04:40:52 dist-test-slave-0kls krb5kdc[26450](info): commencing operation
krb5kdc: starting...
W20260811 04:40:53.131012 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:53 dist-test-slave-0kls krb5kdc[26450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:53.257062 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:32985
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:32985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:53.368484 26464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:53.368713 26464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:53.368752 26464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:53.372761 26464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:53.372819 26464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:53.372838 26464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:53.372855 26464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:53.372870 26464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:53.378137 26464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:32985
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:32985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:53.379215 26464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:53.380261 26464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:53.389086 26476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:53.389071 26473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:53.389106 26464 server_base.cc:1034] running on GCE node
W20260811 04:40:53.389091 26471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:53.389989 26464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:53.391016 26464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:53.392179 26464 hybrid_clock.cc:648] HybridClock initialized: now 1786423253392160 us; error 28 us; skew 500 ppm
Aug 11 04:40:53 dist-test-slave-0kls krb5kdc[26450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423253, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:53.394999 26464 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:53.395927 26464 webserver.cc:460] Webserver started at http://127.2.238.190:45355/ using document root <none> and password file <none>
I20260811 04:40:53.396489 26464 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:53.396551 26464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:53.396795 26464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:53.398870 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "c736ef42b4da4dfe8ecb3d9f4925015c"
format_stamp: "Formatted at 2026-08-11 04:40:53 on dist-test-slave-0kls"
server_key: "984ac926a46a21013a1dd028d871e302"
server_key_iv: "36bf97de67f88e89d33fed2f6024feb0"
server_key_version: "encryptionkey@0"
I20260811 04:40:53.399432 26464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "c736ef42b4da4dfe8ecb3d9f4925015c"
format_stamp: "Formatted at 2026-08-11 04:40:53 on dist-test-slave-0kls"
server_key: "984ac926a46a21013a1dd028d871e302"
server_key_iv: "36bf97de67f88e89d33fed2f6024feb0"
server_key_version: "encryptionkey@0"
I20260811 04:40:53.403190 26464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:53.405552 26484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:53.406870 26464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:53.407002 26464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "c736ef42b4da4dfe8ecb3d9f4925015c"
format_stamp: "Formatted at 2026-08-11 04:40:53 on dist-test-slave-0kls"
server_key: "984ac926a46a21013a1dd028d871e302"
server_key_iv: "36bf97de67f88e89d33fed2f6024feb0"
server_key_version: "encryptionkey@0"
I20260811 04:40:53.407115 26464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:53.420630 26464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:53.423591 26464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:53.423795 26464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:53.430907 26464 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:32985
I20260811 04:40:53.430932 26543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:32985 every 8 connection(s)
I20260811 04:40:53.431800 26464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:53.433728 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26464
I20260811 04:40:53.433900 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:53.434208 3002 external_mini_cluster.cc:1428] Setting key b260e30c8e400b2b1037fa02f25bc928
I20260811 04:40:53.434618 26544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:53.440608 26544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: Bootstrap starting.
I20260811 04:40:53.442999 26544 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:53.443009 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:53.435542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58333 (local address 127.2.238.190:32985)
0811 04:40:53.435992 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:53.436006 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:53.436041 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:40:53.436673 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:53.436689 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:53.437008 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:53.437333 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:53.437741 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:53.438783 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:53.439685 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:53.440211 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:53.440391 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:53.441315 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:53.441320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:53.441334 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:53.441720 (+ 386us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:53.441772 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:53.441844 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:53.441993 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:53.442177 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 04:40:53.442312 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":148,"threads_started":1}
I20260811 04:40:53.443948 26544 log.cc:824] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:53.446293 26544 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: No bootstrap required, opened a new log
I20260811 04:40:53.446313 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:53.450989 26544 raft_consensus.cc:348] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } }
I20260811 04:40:53.451288 26544 raft_consensus.cc:374] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:53.451365 26544 raft_consensus.cc:729] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c736ef42b4da4dfe8ecb3d9f4925015c, State: Initialized, Role: FOLLOWER
I20260811 04:40:53.451819 26544 consensus_queue.cc:260] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } }
I20260811 04:40:53.451992 26544 raft_consensus.cc:388] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:53.452075 26544 raft_consensus.cc:482] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:53.452222 26544 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:53.453248 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:53.448563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42185 (local address 127.2.238.190:32985)
0811 04:40:53.448698 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:53.448705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:53.449000 (+ 295us) server_negotiation.cc:407] Connection header received
0811 04:40:53.449133 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:53.449136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:53.449173 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:53.449237 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:53.449668 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:53.450264 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:53.451351 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:53.451705 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:53.451799 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:53.452723 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:53.452725 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:53.452727 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:53.452739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:53.452763 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:53.452773 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:53.452842 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:53.453008 (+ 166us) server_negotiation.cc:299] Negotiation successful
0811 04:40:53.453060 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260811 04:40:53.453473 26544 raft_consensus.cc:504] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } }
I20260811 04:40:53.453899 26544 leader_election.cc:302] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c736ef42b4da4dfe8ecb3d9f4925015c; no voters:
I20260811 04:40:53.454288 26544 leader_election.cc:288] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:53.454360 26555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:53.454579 26555 raft_consensus.cc:686] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [term 1 LEADER]: Becoming Leader. State: Replica: c736ef42b4da4dfe8ecb3d9f4925015c, State: Running, Role: LEADER
I20260811 04:40:53.454902 26555 consensus_queue.cc:237] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } }
I20260811 04:40:53.455668 26544 sys_catalog.cc:564] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:53.456630 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } } }
I20260811 04:40:53.457284 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:53.457177 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c736ef42b4da4dfe8ecb3d9f4925015c. Latest consensus state: current_term: 1 leader_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c736ef42b4da4dfe8ecb3d9f4925015c" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 32985 } } }
I20260811 04:40:53.457635 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:53.460491 26570 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:53.460583 26570 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:53.460655 26568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:53.461352 26568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:53.467079 26568 catalog_manager.cc:1347] Generated new cluster ID: ba2547d5b8124407b48ea779159b79f0
I20260811 04:40:53.467157 26568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:53.485229 26568 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:53.486218 26568 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:53.496464 26568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c736ef42b4da4dfe8ecb3d9f4925015c: Generated new TSK 0
I20260811 04:40:53.497155 26568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:53.506184 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26464
W20260811 04:40:53.513084 26448 connection.cc:535] client connection to 127.2.238.190:32985 recv error: Network error: failed to read from TLS socket (remote: 127.2.238.190:32985): Connection reset by peer (error 104)
2026-08-11T04:40:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:53 dist-test-slave-0kls krb5kdc[26578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:53 dist-test-slave-0kls krb5kdc[26578](info): set up 2 sockets
Aug 11 04:40:53 dist-test-slave-0kls krb5kdc[26578](info): commencing operation
krb5kdc: starting...
W20260811 04:40:54.576629 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:54 dist-test-slave-0kls krb5kdc[26578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:54.702215 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43423
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:42913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:43423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:54.813205 26595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:54.813464 26595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:54.813508 26595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:54.817438 26595 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:54.817508 26595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:54.817528 26595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:54.817544 26595 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:54.817577 26595 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:54.822726 26595 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:42913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:43423
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:54.823696 26595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:54.824688 26595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:54.833087 26607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:54.833235 26595 server_base.cc:1034] running on GCE node
W20260811 04:40:54.833185 26600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:54.833117 26609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:54.833961 26595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:54.834995 26595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:54.836179 26595 hybrid_clock.cc:648] HybridClock initialized: now 1786423254836139 us; error 50 us; skew 500 ppm
Aug 11 04:40:54 dist-test-slave-0kls krb5kdc[26578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423254, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:54.838973 26595 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:54.839906 26595 webserver.cc:460] Webserver started at http://127.2.238.190:45871/ using document root <none> and password file <none>
I20260811 04:40:54.840476 26595 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:54.840570 26595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:54.840816 26595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:54.842936 26595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "0b1fd9e3daa74280818f8476ac697e22"
format_stamp: "Formatted at 2026-08-11 04:40:54 on dist-test-slave-0kls"
server_key: "8ada8868b9e8575480812ff896fc773c"
server_key_iv: "97281f6f1bc1f6dbe63d3a467c80c2ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:54.843508 26595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "0b1fd9e3daa74280818f8476ac697e22"
format_stamp: "Formatted at 2026-08-11 04:40:54 on dist-test-slave-0kls"
server_key: "8ada8868b9e8575480812ff896fc773c"
server_key_iv: "97281f6f1bc1f6dbe63d3a467c80c2ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:54.847231 26595 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:40:54.849550 26617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:54.850844 26595 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:40:54.850993 26595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "0b1fd9e3daa74280818f8476ac697e22"
format_stamp: "Formatted at 2026-08-11 04:40:54 on dist-test-slave-0kls"
server_key: "8ada8868b9e8575480812ff896fc773c"
server_key_iv: "97281f6f1bc1f6dbe63d3a467c80c2ee"
server_key_version: "encryptionkey@0"
I20260811 04:40:54.851105 26595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:54.855669 26595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:54.858505 26595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:54.858718 26595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:54.865995 26595 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:43423
I20260811 04:40:54.866024 26676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:43423 every 8 connection(s)
I20260811 04:40:54.866921 26595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:54.868314 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26595
I20260811 04:40:54.868469 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:54.868774 3002 external_mini_cluster.cc:1428] Setting key a0f0a24293c27d7eaaab05d2bcd65d16
I20260811 04:40:54.870015 26677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:54.874025 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:54.870198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54705 (local address 127.2.238.190:43423)
0811 04:40:54.870625 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:54.870639 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:54.870686 (+ 47us) server_negotiation.cc:407] Connection header received
0811 04:40:54.871311 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:54.871325 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:54.871601 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:54.871949 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:54.872776 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:54.872781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:54.872793 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:54.872804 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:54.872861 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:54.872881 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:54.873033 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:54.873106 (+ 73us) server_negotiation.cc:299] Negotiation successful
0811 04:40:54.873265 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260811 04:40:54.876194 26677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22: Bootstrap starting.
I20260811 04:40:54.876833 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:54.878741 26677 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:54.879663 26677 log.cc:824] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:54.880769 26680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:54.878925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50155 (local address 127.2.238.190:43423)
0811 04:40:54.879067 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:54.879071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:54.879312 (+ 241us) server_negotiation.cc:407] Connection header received
0811 04:40:54.879405 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:54.879408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:54.879451 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:54.879545 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:54.880297 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:54.880301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:54.880303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:54.880318 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:54.880349 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:54.880366 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:54.880467 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:54.880579 (+ 112us) server_negotiation.cc:299] Negotiation successful
0811 04:40:54.880612 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:40:54.882145 26677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22: No bootstrap required, opened a new log
I20260811 04:40:54.886176 26677 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } }
I20260811 04:40:54.886440 26677 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:54.886529 26677 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b1fd9e3daa74280818f8476ac697e22, State: Initialized, Role: FOLLOWER
I20260811 04:40:54.887041 26677 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } }
I20260811 04:40:54.887223 26677 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:54.887318 26677 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:54.887427 26677 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:54.888414 26677 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } }
I20260811 04:40:54.888780 26677 leader_election.cc:302] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b1fd9e3daa74280818f8476ac697e22; no voters:
I20260811 04:40:54.889191 26677 leader_election.cc:288] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:54.889308 26687 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:54.889591 26687 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [term 1 LEADER]: Becoming Leader. State: Replica: 0b1fd9e3daa74280818f8476ac697e22, State: Running, Role: LEADER
I20260811 04:40:54.889990 26687 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } }
I20260811 04:40:54.890787 26677 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:54.891916 26688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b1fd9e3daa74280818f8476ac697e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } } }
I20260811 04:40:54.892068 26688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:54.892366 26690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b1fd9e3daa74280818f8476ac697e22. Latest consensus state: current_term: 1 leader_uuid: "0b1fd9e3daa74280818f8476ac697e22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b1fd9e3daa74280818f8476ac697e22" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43423 } } }
I20260811 04:40:54.892472 26699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:54.892563 26690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:54.895840 26699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:54.901772 26699 catalog_manager.cc:1347] Generated new cluster ID: 471fdfd1b9974130a9798706d1a7c4fa
I20260811 04:40:54.901904 26699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:54.908751 26699 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:54.909036 26699 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:54.913587 26699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b1fd9e3daa74280818f8476ac697e22: Generated new TSK 0
I20260811 04:40:54.914271 26699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:54.923893 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26595
2026-08-11T04:40:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1419 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:54 dist-test-slave-0kls krb5kdc[26711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:54 dist-test-slave-0kls krb5kdc[26711](info): set up 2 sockets
Aug 11 04:40:54 dist-test-slave-0kls krb5kdc[26711](info): commencing operation
krb5kdc: starting...
W20260811 04:40:55.997313 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:56.128696 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39695
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:35921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:39695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:56.238749 26725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:56.239032 26725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:56.239120 26725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:56.243098 26725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:56.243181 26725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:56.243228 26725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:56.243261 26725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:56.243295 26725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:56.248358 26725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:35921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:39695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:39695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:56.249396 26725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:56.250502 26725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:56.258966 26731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:56.258986 26733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:56.259024 26725 server_base.cc:1034] running on GCE node
W20260811 04:40:56.259037 26730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:56.259729 26725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:56.260825 26725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:56.261984 26725 hybrid_clock.cc:648] HybridClock initialized: now 1786423256261970 us; error 27 us; skew 500 ppm
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423256, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:56.264729 26725 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:56.265623 26725 webserver.cc:460] Webserver started at http://127.2.238.190:40629/ using document root <none> and password file <none>
I20260811 04:40:56.266220 26725 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:56.266284 26725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:56.266453 26725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:56.268436 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "fa58d42aac41426aa8373fbebf690951"
format_stamp: "Formatted at 2026-08-11 04:40:56 on dist-test-slave-0kls"
server_key: "6e40e482012e837c5ce427eb769eef44"
server_key_iv: "518471f9c36847fa3680290b172b472a"
server_key_version: "encryptionkey@0"
I20260811 04:40:56.268954 26725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "fa58d42aac41426aa8373fbebf690951"
format_stamp: "Formatted at 2026-08-11 04:40:56 on dist-test-slave-0kls"
server_key: "6e40e482012e837c5ce427eb769eef44"
server_key_iv: "518471f9c36847fa3680290b172b472a"
server_key_version: "encryptionkey@0"
I20260811 04:40:56.272652 26725 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:40:56.275050 26742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:56.276293 26725 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:40:56.276396 26725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "fa58d42aac41426aa8373fbebf690951"
format_stamp: "Formatted at 2026-08-11 04:40:56 on dist-test-slave-0kls"
server_key: "6e40e482012e837c5ce427eb769eef44"
server_key_iv: "518471f9c36847fa3680290b172b472a"
server_key_version: "encryptionkey@0"
I20260811 04:40:56.276477 26725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:56.288482 26725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:56.291821 26725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:56.291978 26725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:56.299775 26725 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:39695
I20260811 04:40:56.299817 26797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:39695 every 8 connection(s)
I20260811 04:40:56.300740 26725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:56.303622 26798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:56.305271 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26725
I20260811 04:40:56.305408 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:56.305719 3002 external_mini_cluster.cc:1428] Setting key 446acea82b04a95676ce0dc15cb4c56e
I20260811 04:40:56.310019 26798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: Bootstrap starting.
I20260811 04:40:56.312242 26798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:56.312978 26798 log.cc:824] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:40:56.315017 26798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: No bootstrap required, opened a new log
I20260811 04:40:56.318799 26798 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } }
I20260811 04:40:56.319092 26798 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:56.319173 26798 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa58d42aac41426aa8373fbebf690951, State: Initialized, Role: FOLLOWER
I20260811 04:40:56.319674 26798 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } }
I20260811 04:40:56.319778 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:56.307236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49643 (local address 127.2.238.190:39695)
0811 04:40:56.307684 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:56.307698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:56.307734 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:40:56.308361 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:56.308377 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:56.308704 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:56.309036 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:56.309983 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:56.310982 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:56.311793 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:56.312266 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:56.312435 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:56.315020 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:56.315025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:56.315038 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:56.315065 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:56.316763 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:56.317211 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:56.317219 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:56.317230 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:56.317338 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:56.317635 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:56.317638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:56.317640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:56.318103 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:56.318338 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:56.318592 (+ 254us) server_negotiation.cc:299] Negotiation successful
0811 04:40:56.318904 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":132,"threads_started":1}
I20260811 04:40:56.319844 26798 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:56.319941 26798 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:56.320045 26798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:56.321022 26798 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } }
I20260811 04:40:56.321385 26798 leader_election.cc:302] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa58d42aac41426aa8373fbebf690951; no voters:
I20260811 04:40:56.321810 26798 leader_election.cc:288] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:56.322177 26805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:56.322454 26805 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [term 1 LEADER]: Becoming Leader. State: Replica: fa58d42aac41426aa8373fbebf690951, State: Running, Role: LEADER
I20260811 04:40:56.322824 26805 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } }
I20260811 04:40:56.322891 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:56.323195 26798 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:56.325423 26806 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa58d42aac41426aa8373fbebf690951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } } }
I20260811 04:40:56.325554 26806 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:56.326234 26807 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa58d42aac41426aa8373fbebf690951. Latest consensus state: current_term: 1 leader_uuid: "fa58d42aac41426aa8373fbebf690951" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa58d42aac41426aa8373fbebf690951" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 39695 } } }
I20260811 04:40:56.326468 26807 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951 [sys.catalog]: This master's current role is: LEADER
W20260811 04:40:56.328801 26824 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:40:56.328894 26824 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:40:56.328981 26826 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:56.329937 26826 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:56.335695 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:56.325173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58277 (local address 127.2.238.190:39695)
0811 04:40:56.325846 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:56.325851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:56.326512 (+ 661us) server_negotiation.cc:407] Connection header received
0811 04:40:56.326653 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:56.326656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:56.326709 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:56.326784 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:56.327684 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:56.328475 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:56.329594 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:56.329989 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:56.330104 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:56.331856 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:56.331860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:56.331863 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:40:56.331892 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:56.333722 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:56.334282 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:56.334289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:56.334290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:56.334387 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:40:56.334726 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:40:56.334729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:40:56.334731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:40:56.334970 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:56.335104 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:56.335400 (+ 296us) server_negotiation.cc:299] Negotiation successful
0811 04:40:56.335526 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567}
I20260811 04:40:56.337464 26826 catalog_manager.cc:1347] Generated new cluster ID: 39788f1dfc2641c79dd929730be00541
I20260811 04:40:56.337533 26826 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:56.347597 26826 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:56.349071 26826 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:56.361919 26826 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa58d42aac41426aa8373fbebf690951: Generated new TSK 0
I20260811 04:40:56.362636 26826 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:56.372695 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26725
2026-08-11T04:40:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1448 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26833](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26833](info): set up 2 sockets
Aug 11 04:40:56 dist-test-slave-0kls krb5kdc[26833](info): commencing operation
krb5kdc: starting...
W20260811 04:40:57.438288 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:57 dist-test-slave-0kls krb5kdc[26833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:57.563001 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42151
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:40599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:42151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:57.672281 26848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:57.672518 26848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:57.672556 26848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:57.676535 26848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:57.676594 26848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:57.676612 26848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:57.676630 26848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:57.676643 26848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:57.681855 26848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:40599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:42151
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:42151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:57.682845 26848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:57.683835 26848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:57.692545 26857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:57.692530 26854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:57.692577 26848 server_base.cc:1034] running on GCE node
W20260811 04:40:57.692516 26855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:57.693295 26848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:57.694324 26848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:57.695503 26848 hybrid_clock.cc:648] HybridClock initialized: now 1786423257695483 us; error 30 us; skew 500 ppm
Aug 11 04:40:57 dist-test-slave-0kls krb5kdc[26833](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423257, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:57.703714 26848 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:57.704882 26848 webserver.cc:460] Webserver started at http://127.2.238.190:38113/ using document root <none> and password file <none>
I20260811 04:40:57.705480 26848 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:57.705576 26848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:57.705858 26848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:57.708093 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "06c42d4b8ca94be38a41438229a190f5"
format_stamp: "Formatted at 2026-08-11 04:40:57 on dist-test-slave-0kls"
server_key: "aeb899e00fd199ba6b049d5cdc954340"
server_key_iv: "b584963eeb0e30bda88c54bef3d4894a"
server_key_version: "encryptionkey@0"
I20260811 04:40:57.708698 26848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "06c42d4b8ca94be38a41438229a190f5"
format_stamp: "Formatted at 2026-08-11 04:40:57 on dist-test-slave-0kls"
server_key: "aeb899e00fd199ba6b049d5cdc954340"
server_key_iv: "b584963eeb0e30bda88c54bef3d4894a"
server_key_version: "encryptionkey@0"
I20260811 04:40:57.712733 26848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260811 04:40:57.715256 26864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:57.716475 26848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:40:57.716629 26848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "06c42d4b8ca94be38a41438229a190f5"
format_stamp: "Formatted at 2026-08-11 04:40:57 on dist-test-slave-0kls"
server_key: "aeb899e00fd199ba6b049d5cdc954340"
server_key_iv: "b584963eeb0e30bda88c54bef3d4894a"
server_key_version: "encryptionkey@0"
I20260811 04:40:57.716742 26848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:57.746171 26848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:57.749574 26848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:57.749830 26848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:57.756633 26848 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:42151
I20260811 04:40:57.756749 26924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:42151 every 8 connection(s)
I20260811 04:40:57.757704 26848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:57.758967 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26848
I20260811 04:40:57.759121 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:57.759498 3002 external_mini_cluster.cc:1428] Setting key 8492b3ca25fbb390412eb776f6bf696a
I20260811 04:40:57.761508 26925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:57.767441 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5: Bootstrap starting.
I20260811 04:40:57.768779 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:57.761066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55811 (local address 127.2.238.190:42151)
0811 04:40:57.761609 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:57.761631 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:57.761682 (+ 51us) server_negotiation.cc:407] Connection header received
0811 04:40:57.762394 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:57.762410 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:57.762755 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:57.763103 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:57.763536 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:57.764487 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:57.765371 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:57.765977 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:57.766162 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:57.767048 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:57.767052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:57.767066 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:57.767465 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:57.767553 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:57.767593 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:57.767701 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:57.767854 (+ 153us) server_negotiation.cc:299] Negotiation successful
0811 04:40:57.768067 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":173,"threads_started":1}
I20260811 04:40:57.769937 26925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:57.770900 26925 log.cc:824] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:57.771978 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:57.773053 26925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5: No bootstrap required, opened a new log
I20260811 04:40:57.778053 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:57.774031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55035 (local address 127.2.238.190:42151)
0811 04:40:57.774156 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:57.774160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:57.774383 (+ 223us) server_negotiation.cc:407] Connection header received
0811 04:40:57.774488 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:57.774490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:57.774523 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:57.774586 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:57.774848 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:57.775356 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:57.776510 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:40:57.776723 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:40:57.776800 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:40:57.777603 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:57.777605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:57.777607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:57.777617 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:57.777640 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:57.777650 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:57.777722 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:57.777860 (+ 138us) server_negotiation.cc:299] Negotiation successful
0811 04:40:57.777916 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 04:40:57.777873 26925 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } }
I20260811 04:40:57.778249 26925 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:57.778328 26925 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06c42d4b8ca94be38a41438229a190f5, State: Initialized, Role: FOLLOWER
I20260811 04:40:57.778811 26925 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } }
I20260811 04:40:57.779019 26925 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:57.779115 26925 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:57.779214 26925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:57.780076 26925 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } }
I20260811 04:40:57.780419 26925 leader_election.cc:302] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06c42d4b8ca94be38a41438229a190f5; no voters:
I20260811 04:40:57.780682 26925 leader_election.cc:288] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:57.780835 26936 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:57.781090 26936 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [term 1 LEADER]: Becoming Leader. State: Replica: 06c42d4b8ca94be38a41438229a190f5, State: Running, Role: LEADER
I20260811 04:40:57.781399 26936 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } }
I20260811 04:40:57.782022 26925 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:57.783293 26938 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06c42d4b8ca94be38a41438229a190f5. Latest consensus state: current_term: 1 leader_uuid: "06c42d4b8ca94be38a41438229a190f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } } }
I20260811 04:40:57.783282 26937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06c42d4b8ca94be38a41438229a190f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06c42d4b8ca94be38a41438229a190f5" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 42151 } } }
I20260811 04:40:57.783433 26938 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:57.783433 26937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:57.783862 26945 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:57.786536 26945 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:57.791939 26945 catalog_manager.cc:1347] Generated new cluster ID: 55ff7a32bb2d4c75b1f0649ce1a9d4d5
I20260811 04:40:57.792001 26945 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:57.808604 26945 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:57.809962 26945 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:57.815987 26945 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06c42d4b8ca94be38a41438229a190f5: Generated new TSK 0
I20260811 04:40:57.816730 26945 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:57.825307 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26848
2026-08-11T04:40:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1453 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:57 dist-test-slave-0kls krb5kdc[26958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:57 dist-test-slave-0kls krb5kdc[26958](info): set up 2 sockets
Aug 11 04:40:57 dist-test-slave-0kls krb5kdc[26958](info): commencing operation
krb5kdc: starting...
W20260811 04:40:58.910861 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:40:58 dist-test-slave-0kls krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:40:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:40:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:40:59.037381 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43405
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:38121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:40:59.147764 26971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:40:59.148066 26971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:40:59.148141 26971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:40:59.152086 26971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:40:59.152172 26971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:40:59.152210 26971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:40:59.152251 26971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:40:59.152281 26971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:40:59.157392 26971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:38121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:43405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:40:59.158522 26971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:40:59.159540 26971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:40:59.168128 26977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:59.168118 26979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:40:59.168119 26976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:40:59.168320 26971 server_base.cc:1034] running on GCE node
I20260811 04:40:59.169014 26971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:40:59.170083 26971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:40:59.171240 26971 hybrid_clock.cc:648] HybridClock initialized: now 1786423259171216 us; error 35 us; skew 500 ppm
Aug 11 04:40:59 dist-test-slave-0kls krb5kdc[26958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423259, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:40:59.174361 26971 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:40:59.175271 26971 webserver.cc:460] Webserver started at http://127.2.238.190:41213/ using document root <none> and password file <none>
I20260811 04:40:59.175814 26971 fs_manager.cc:359] Metadata directory not provided
I20260811 04:40:59.175900 26971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:40:59.176105 26971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:40:59.178319 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569"
format_stamp: "Formatted at 2026-08-11 04:40:59 on dist-test-slave-0kls"
server_key: "ac1982d64f5c4e4f97e4b3e2cec5ba3a"
server_key_iv: "b9ae6f82ffa4ad60752d9d3e38738815"
server_key_version: "encryptionkey@0"
I20260811 04:40:59.178882 26971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569"
format_stamp: "Formatted at 2026-08-11 04:40:59 on dist-test-slave-0kls"
server_key: "ac1982d64f5c4e4f97e4b3e2cec5ba3a"
server_key_iv: "b9ae6f82ffa4ad60752d9d3e38738815"
server_key_version: "encryptionkey@0"
I20260811 04:40:59.182703 26971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:40:59.185163 26989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:40:59.186486 26971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:40:59.186632 26971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569"
format_stamp: "Formatted at 2026-08-11 04:40:59 on dist-test-slave-0kls"
server_key: "ac1982d64f5c4e4f97e4b3e2cec5ba3a"
server_key_iv: "b9ae6f82ffa4ad60752d9d3e38738815"
server_key_version: "encryptionkey@0"
I20260811 04:40:59.186748 26971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:40:59.191494 26971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:40:59.194350 26971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:40:59.194537 26971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:40:59.201375 26971 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:43405
I20260811 04:40:59.201433 27048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:43405 every 8 connection(s)
I20260811 04:40:59.202431 26971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:40:59.203542 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 26971
I20260811 04:40:59.203688 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:40:59.204002 3002 external_mini_cluster.cc:1428] Setting key 8633a8fc65766465bdce99c8e4ef9010
I20260811 04:40:59.206019 27050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:40:59.209280 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:59.205410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46087 (local address 127.2.238.190:43405)
0811 04:40:59.205831 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:59.205853 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:59.205889 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:40:59.206576 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:59.206593 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:59.206874 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:59.207164 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:59.208028 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:59.208032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:59.208044 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:59.208054 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:59.208106 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:59.208124 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:59.208249 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:59.208362 (+ 113us) server_negotiation.cc:299] Negotiation successful
0811 04:40:59.208518 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":146,"threads_started":1}
I20260811 04:40:59.212291 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:40:59.212685 27050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569: Bootstrap starting.
I20260811 04:40:59.215593 27050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569: Neither blocks nor log segments found. Creating new log.
I20260811 04:40:59.216403 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:40:59.214668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52173 (local address 127.2.238.190:43405)
0811 04:40:59.214794 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 04:40:59.214798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:40:59.215046 (+ 248us) server_negotiation.cc:407] Connection header received
0811 04:40:59.215174 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:40:59.215176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:40:59.215207 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:40:59.215263 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:40:59.215984 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:40:59.215987 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:40:59.215988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0811 04:40:59.215999 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:40:59.216024 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0811 04:40:59.216033 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:40:59.216099 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 04:40:59.216228 (+ 129us) server_negotiation.cc:299] Negotiation successful
0811 04:40:59.216260 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260811 04:40:59.216611 27050 log.cc:824] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569: Log is configured to *not* fsync() on all Append() calls
I20260811 04:40:59.218672 27050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569: No bootstrap required, opened a new log
I20260811 04:40:59.222241 27050 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } }
I20260811 04:40:59.222482 27050 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260811 04:40:59.222561 27050 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a6dccca7424bb1bf8f9dc9e1e41569, State: Initialized, Role: FOLLOWER
I20260811 04:40:59.223071 27050 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } }
I20260811 04:40:59.223232 27050 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260811 04:40:59.223302 27050 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260811 04:40:59.223398 27050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:40:59.224301 27050 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } }
I20260811 04:40:59.224659 27050 leader_election.cc:302] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a6dccca7424bb1bf8f9dc9e1e41569; no voters:
I20260811 04:40:59.224987 27050 leader_election.cc:288] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260811 04:40:59.225149 27062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:40:59.225381 27062 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [term 1 LEADER]: Becoming Leader. State: Replica: b2a6dccca7424bb1bf8f9dc9e1e41569, State: Running, Role: LEADER
I20260811 04:40:59.225775 27062 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } }
I20260811 04:40:59.226346 27050 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:40:59.227586 27064 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a6dccca7424bb1bf8f9dc9e1e41569. Latest consensus state: current_term: 1 leader_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } } }
I20260811 04:40:59.227720 27064 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:59.228127 27072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:40:59.228145 27063 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a6dccca7424bb1bf8f9dc9e1e41569" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 43405 } } }
I20260811 04:40:59.228281 27063 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569 [sys.catalog]: This master's current role is: LEADER
I20260811 04:40:59.231245 27072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:40:59.236933 27072 catalog_manager.cc:1347] Generated new cluster ID: 25f825cec8cf4b02ae8580942ab55cc8
I20260811 04:40:59.237022 27072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:40:59.252517 27072 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:40:59.252878 27072 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:40:59.268785 27072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2a6dccca7424bb1bf8f9dc9e1e41569: Generated new TSK 0
I20260811 04:40:59.269685 27072 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260811 04:40:59.281862 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 26971
2026-08-11T04:40:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1455 ms)
[----------] 18 tests from AuthTokenIssuingTest (26046 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:40:59 dist-test-slave-0kls krb5kdc[27085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:40:59 dist-test-slave-0kls krb5kdc[27085](info): set up 2 sockets
Aug 11 04:40:59 dist-test-slave-0kls krb5kdc[27085](info): commencing operation
krb5kdc: starting...
W20260811 04:41:00.353436 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:41:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:41:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:41:00.480612 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41629
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:41:00.588016 27099 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:00.588330 27099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:41:00.588421 27099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:00.588501 27099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:41:00.592489 27099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:00.592660 27099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:00.592705 27099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:00.592764 27099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:00.592806 27099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:00.598095 27099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:41629
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:00.599134 27099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:00.600700 27099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:00.609061 27107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:00.609105 27110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:00.609078 27106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:00.609398 27099 server_base.cc:1034] running on GCE node
I20260811 04:41:00.610074 27099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:00.611127 27099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:00.612287 27099 hybrid_clock.cc:648] HybridClock initialized: now 1786423260612270 us; error 28 us; skew 500 ppm
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:00.614912 27099 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:41:00.615727 27099 webserver.cc:460] Webserver started at http://127.2.238.190:41411/ using document root <none> and password file <none>
I20260811 04:41:00.616251 27099 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:00.616312 27099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:00.616557 27099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:00.618688 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "d0f694b5cffa49459b2cdc3489e030bc"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "56002dbadca8389033cf20a365dd9671"
server_key_iv: "6e47c82c82f5cdf70e922952c0c36718"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.619292 27099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "d0f694b5cffa49459b2cdc3489e030bc"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "56002dbadca8389033cf20a365dd9671"
server_key_iv: "6e47c82c82f5cdf70e922952c0c36718"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.623466 27099 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260811 04:41:00.625962 27117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:00.627275 27099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:41:00.627415 27099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "d0f694b5cffa49459b2cdc3489e030bc"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "56002dbadca8389033cf20a365dd9671"
server_key_iv: "6e47c82c82f5cdf70e922952c0c36718"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.627526 27099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:00.649327 27099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:00.652055 27099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:00.652267 27099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:00.659353 27099 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:41629
I20260811 04:41:00.659651 27177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:41629 every 8 connection(s)
I20260811 04:41:00.660390 27099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:41:00.663158 27178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:00.666831 27178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:00.667809 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27099
I20260811 04:41:00.667968 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:41:00.668241 3002 external_mini_cluster.cc:1428] Setting key 7c2a0790f68212ba19e50a894ff7bc5b
WARNING: no policy specified for kudu/127.2.238.189@KRBTEST.COM; defaulting to no policy
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.189@KRBTEST.COM" created.
I20260811 04:41:00.682760 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.669206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41629 (local address 127.0.0.1:57772)
0811 04:41:00.669761 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.669845 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:00.670871 (+ 1026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:00.671542 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:00.671547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:00.671977 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:00.672218 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.672230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.673405 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.673408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.674188 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.674199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.674833 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.674836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.674983 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.675576 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:00.675593 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:00.677257 (+ 1664us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:00.679394 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.679404 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.679414 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.679744 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.680165 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.680167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.680169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.680293 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.681056 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:00.681060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:00.681184 (+ 124us) client_negotiation.cc:769] Sending connection context
0811 04:41:00.681368 (+ 184us) client_negotiation.cc:240] Negotiation successful
0811 04:41:00.681720 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":101,"thread_start_us":128,"threads_started":1}
I20260811 04:41:00.682760 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.669206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57772 (local address 127.2.238.190:41629)
0811 04:41:00.669632 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:00.669653 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:00.669977 (+ 324us) server_negotiation.cc:407] Connection header received
0811 04:41:00.671169 (+ 1192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:00.671173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:00.671361 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:00.671543 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:00.672354 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.673282 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.674306 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.674668 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.674834 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.677408 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:00.677413 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:00.677423 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:00.677458 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:00.679269 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.679906 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.679913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.679914 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.680062 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.680391 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.680397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.680402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.680930 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:00.681084 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:00.681431 (+ 347us) server_negotiation.cc:299] Negotiation successful
0811 04:41:00.681671 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260811 04:41:00.685336 27178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:00.686165 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.685903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47066)
0811 04:41:00.686030 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.686102 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 04:41:00.686414 27121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
I20260811 04:41:00.687323 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.687111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47080)
0811 04:41:00.687222 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.687268 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260811 04:41:00.687585 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:35615: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
I20260811 04:41:00.696882 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:35615
--webserver_interface=127.2.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:41:00.736819 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } attempt: 1
I20260811 04:41:00.737761 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.737420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47094)
0811 04:41:00.737595 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.737681 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 04:41:00.738559 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.738350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47110)
0811 04:41:00.738458 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.738505 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:00.738714 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:35615: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
I20260811 04:41:00.806875 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } attempt: 2
I20260811 04:41:00.807732 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.807423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47112)
0811 04:41:00.807591 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.807657 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260811 04:41:00.808521 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.808276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47126)
0811 04:41:00.808387 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.808447 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:00.808725 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:35615: Network error: Client connection negotiation failed: client connection to 127.2.238.189:35615: connect: Connection refused (error 111)
W20260811 04:41:00.811792 27185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:00.812127 27185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:41:00.812193 27185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:00.812273 27185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:41:00.816419 27185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:00.816514 27185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:00.816560 27185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:00.816619 27185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:00.816661 27185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:00.822250 27185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:35615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:00.823442 27185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:00.824564 27185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:00.833331 27193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:00.833343 27192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:00.833331 27195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:00.833654 27185 server_base.cc:1034] running on GCE node
I20260811 04:41:00.834291 27185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:00.835412 27185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:00.836576 27185 hybrid_clock.cc:648] HybridClock initialized: now 1786423260836555 us; error 33 us; skew 500 ppm
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:00.839466 27185 init.cc:377] Logged in from keytab as kudu/127.2.238.189@KRBTEST.COM (short username kudu)
I20260811 04:41:00.840413 27185 webserver.cc:460] Webserver started at http://127.2.238.189:38925/ using document root <none> and password file <none>
I20260811 04:41:00.840987 27185 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:00.841049 27185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:00.841301 27185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:00.843425 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data/instance:
uuid: "099f7c7e1259487bab991a8d42dbf4dd"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "fe90c4b83e1af1b079f68788771f8e08"
server_key_iv: "cdf77e75c2ebe11cad9c149affd2138d"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.844002 27185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal/instance:
uuid: "099f7c7e1259487bab991a8d42dbf4dd"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "fe90c4b83e1af1b079f68788771f8e08"
server_key_iv: "cdf77e75c2ebe11cad9c149affd2138d"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.847756 27185 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260811 04:41:00.850101 27203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:00.851310 27185 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:41:00.851454 27185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "099f7c7e1259487bab991a8d42dbf4dd"
format_stamp: "Formatted at 2026-08-11 04:41:00 on dist-test-slave-0kls"
server_key: "fe90c4b83e1af1b079f68788771f8e08"
server_key_iv: "cdf77e75c2ebe11cad9c149affd2138d"
server_key_version: "encryptionkey@0"
I20260811 04:41:00.851565 27185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:00.879554 27185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:00.883046 27185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:00.883275 27185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:00.890461 27185 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.189:35615
I20260811 04:41:00.890775 27261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.189:35615 every 8 connection(s)
I20260811 04:41:00.891471 27185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
I20260811 04:41:00.894461 27262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:00.894526 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27185
I20260811 04:41:00.894655 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-1/wal/instance
I20260811 04:41:00.894994 3002 external_mini_cluster.cc:1428] Setting key d4baee921430db9a53dcada25d35a422
I20260811 04:41:00.899305 27262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:00.899892 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } attempt: 3
WARNING: no policy specified for kudu/127.2.238.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.188@KRBTEST.COM" created.
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:00.916769 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.900493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47132)
0811 04:41:00.900756 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.900776 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:00.900884 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:00.903062 (+ 2178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:00.903065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:00.903282 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:00.903437 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.903444 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.905105 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.905109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.905616 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.905624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.906333 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.906336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.906434 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.907074 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:00.907086 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:00.912124 (+ 5038us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:00.914731 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.914734 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.914737 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.914972 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.915309 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915311 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.915313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.915435 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.916294 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:00.916297 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:00.916387 (+ 90us) client_negotiation.cc:769] Sending connection context
0811 04:41:00.916483 (+ 96us) client_negotiation.cc:240] Negotiation successful
0811 04:41:00.916637 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260811 04:41:00.917626 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.901437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57780 (local address 127.2.238.190:41629)
0811 04:41:00.901563 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:00.901568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:00.902070 (+ 502us) server_negotiation.cc:407] Connection header received
0811 04:41:00.902903 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:00.902907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:00.902964 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:00.903040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:00.904115 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.904680 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.905741 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.906038 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.906179 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.912877 (+ 6698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:00.912883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:00.912887 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:00.912920 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:00.915313 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915812 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.915815 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.915817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.915898 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.916156 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.916160 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.916162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.916401 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:00.916532 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:00.917249 (+ 717us) server_negotiation.cc:299] Negotiation successful
0811 04:41:00.917393 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:41:00.918318 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.901201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.2.238.189:35615)
0811 04:41:00.901838 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:00.901872 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:00.901912 (+ 40us) server_negotiation.cc:407] Connection header received
0811 04:41:00.902619 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:00.902637 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:00.902867 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:00.903059 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:00.903612 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.904975 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.905745 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.906204 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.906381 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.912281 (+ 5900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:00.912289 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:00.912303 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:00.912334 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:00.914628 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915078 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.915084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.915090 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.915198 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915543 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.915548 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.915552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.916167 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:00.916314 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:00.917061 (+ 747us) server_negotiation.cc:299] Negotiation successful
0811 04:41:00.917934 (+ 873us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":475,"spinlock_wait_cycles":3840,"thread_start_us":120,"threads_started":1}
I20260811 04:41:00.918314 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.901366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41629 (local address 127.0.0.1:57780)
0811 04:41:00.901911 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.901975 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:00.902652 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:00.903238 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:00.903244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:00.903708 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:00.903953 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.903964 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.904805 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.904809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.905615 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.905625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.906221 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.906226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.906377 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.907023 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:00.907040 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:00.912732 (+ 5692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:00.915436 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915445 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.915451 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.915711 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.915982 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.915985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.915987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.916065 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.916547 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:00.916555 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:00.916749 (+ 194us) client_negotiation.cc:769] Sending connection context
0811 04:41:00.916944 (+ 195us) client_negotiation.cc:240] Negotiation successful
0811 04:41:00.917331 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":19,"spinlock_wait_cycles":1280,"thread_start_us":121,"threads_started":1}
I20260811 04:41:00.920523 27262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:00.920557 27178 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:00.921335 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.921078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:41990)
0811 04:41:00.921199 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.921260 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260811 04:41:00.922088 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.921855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42006)
0811 04:41:00.921985 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.922037 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260811 04:41:00.922246 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Aug 11 04:41:00 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:41:00.930099 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.921100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47136)
0811 04:41:00.921392 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.921405 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:00.921487 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:00.921756 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:00.921762 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:00.921980 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:00.922122 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.922129 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.923048 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.923051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.923692 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:00.923701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.924226 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.924229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:00.924368 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.925150 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:00.925162 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:00.927113 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:00.928621 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.928624 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.928625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.928806 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.929053 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:00.929055 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:00.929057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:00.929127 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:00.929512 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:00.929514 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:00.929671 (+ 157us) client_negotiation.cc:769] Sending connection context
0811 04:41:00.929760 (+ 89us) client_negotiation.cc:240] Negotiation successful
0811 04:41:00.929937 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":18}
I20260811 04:41:00.930254 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.921284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.2.238.189:35615)
0811 04:41:00.921422 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:00.921428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:00.921447 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:41:00.921568 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:00.921571 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:00.921622 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:00.922223 (+ 601us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:00.922299 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.922944 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.923817 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:00.924093 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:00.924274 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:00.927221 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:00.927225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:00.927227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:00.927248 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:00.928470 (+ 1222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.928897 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.928900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.928901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.928971 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:00.929241 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:00.929244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:00.929245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:00.929430 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:00.929503 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:00.929864 (+ 361us) server_negotiation.cc:299] Negotiation successful
0811 04:41:00.929990 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":51}
I20260811 04:41:00.930999 27262 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
I20260811 04:41:00.931932 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.931538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42008)
0811 04:41:00.931756 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.931852 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260811 04:41:00.932225 27207 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
I20260811 04:41:00.932765 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:35251
--webserver_interface=127.2.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:33259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:41:00.933086 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.932878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42018)
0811 04:41:00.932987 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.933031 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:00.933243 27262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
I20260811 04:41:00.953419 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 1
I20260811 04:41:00.954371 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.953988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42030)
0811 04:41:00.954193 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.954271 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260811 04:41:00.955269 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.954984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42036)
0811 04:41:00.955145 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.955207 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
W20260811 04:41:00.955435 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
I20260811 04:41:00.982618 27262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 1
I20260811 04:41:00.983567 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.983253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42050)
0811 04:41:00.983419 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.983486 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260811 04:41:00.984301 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:00.984080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42060)
0811 04:41:00.984193 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:00.984248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260811 04:41:00.984437 27262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
I20260811 04:41:01.030584 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 2
I20260811 04:41:01.031450 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.031145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42076)
0811 04:41:01.031306 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.031375 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 04:41:01.032277 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.032077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42082)
0811 04:41:01.032179 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.032226 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260811 04:41:01.032421 27178 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
I20260811 04:41:01.052577 27262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 2
I20260811 04:41:01.053467 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.053144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42098)
0811 04:41:01.053281 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.053360 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260811 04:41:01.054245 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.054015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42108)
0811 04:41:01.054134 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.054186 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260811 04:41:01.054389 27262 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:35251: Network error: Client connection negotiation failed: client connection to 127.2.238.188:35251: connect: Connection refused (error 111)
W20260811 04:41:01.054956 27271 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:01.055265 27271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:41:01.055312 27271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:01.055379 27271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:41:01.059744 27271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:01.059818 27271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:01.059840 27271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:01.059893 27271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:01.059931 27271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:01.065299 27271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:33259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:41629,127.2.238.189:35615,127.2.238.188:35251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:35251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:01.066433 27271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:01.067487 27271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:01.075744 27277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:01.075765 27280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:01.075790 27276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:01.075901 27271 server_base.cc:1034] running on GCE node
I20260811 04:41:01.076628 27271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:01.077643 27271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:01.078836 27271 hybrid_clock.cc:648] HybridClock initialized: now 1786423261078816 us; error 29 us; skew 500 ppm
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:01.081553 27271 init.cc:377] Logged in from keytab as kudu/127.2.238.188@KRBTEST.COM (short username kudu)
I20260811 04:41:01.082572 27271 webserver.cc:460] Webserver started at http://127.2.238.188:36163/ using document root <none> and password file <none>
I20260811 04:41:01.083181 27271 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:01.083307 27271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:01.083544 27271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:01.085537 27271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data/instance:
uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b"
format_stamp: "Formatted at 2026-08-11 04:41:01 on dist-test-slave-0kls"
server_key: "b806e7a515c1b04fe850d43509aa7a48"
server_key_iv: "4453765b29220482842bc7f8788f6ede"
server_key_version: "encryptionkey@0"
I20260811 04:41:01.086095 27271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal/instance:
uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b"
format_stamp: "Formatted at 2026-08-11 04:41:01 on dist-test-slave-0kls"
server_key: "b806e7a515c1b04fe850d43509aa7a48"
server_key_iv: "4453765b29220482842bc7f8788f6ede"
server_key_version: "encryptionkey@0"
I20260811 04:41:01.090065 27271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260811 04:41:01.092365 27288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:01.093453 27271 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260811 04:41:01.093585 27271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b"
format_stamp: "Formatted at 2026-08-11 04:41:01 on dist-test-slave-0kls"
server_key: "b806e7a515c1b04fe850d43509aa7a48"
server_key_iv: "4453765b29220482842bc7f8788f6ede"
server_key_version: "encryptionkey@0"
I20260811 04:41:01.093686 27271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:01.109117 27271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:01.111873 27271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:01.112059 27271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:01.119000 27271 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.188:35251
I20260811 04:41:01.119302 27348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.188:35251 every 8 connection(s)
I20260811 04:41:01.119995 27271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
I20260811 04:41:01.120538 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27271
I20260811 04:41:01.120671 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786422982503306-3002-0/minicluster-data/master-2/wal/instance
I20260811 04:41:01.120949 3002 external_mini_cluster.cc:1428] Setting key 922ccd8f3feb9a65c27afe1f23805062
I20260811 04:41:01.123207 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:01.127552 27349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:01.131565 27178 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 3
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:01.137079 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.122256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.2.238.188:35251)
0811 04:41:01.122648 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.122663 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.122698 (+ 35us) server_negotiation.cc:407] Connection header received
0811 04:41:01.123418 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.123435 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.123745 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.124004 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.124883 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.126021 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.127231 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.127711 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.127820 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.130768 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.130776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.130790 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.130815 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.133450 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.133943 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.133949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.133955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.134071 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.134414 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.134420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.134425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.134940 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.135150 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.135704 (+ 554us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.135963 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":127,"threads_started":1}
I20260811 04:41:01.139715 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.129603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.2.238.190:41629)
0811 04:41:01.129756 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.129760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.130107 (+ 347us) server_negotiation.cc:407] Connection header received
0811 04:41:01.130274 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.130277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.130318 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.130380 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.131277 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.131917 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.133106 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.133309 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.133409 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.135982 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.135985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.135987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.136011 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.137711 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.138219 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.138223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.138224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.138310 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.138607 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.138612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.138613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.138911 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.139013 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.139429 (+ 416us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.139546 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260811 04:41:01.139734 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.129551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:41629 (local address 127.0.0.1:57790)
0811 04:41:01.129973 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.130023 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:01.130157 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:01.130467 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:01.130475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:01.130862 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:01.131157 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.131164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.132103 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.132108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.132995 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.133004 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.133414 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.133417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.133534 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.134248 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:01.134278 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:01.135839 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:01.137844 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.137853 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.137858 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.138103 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.138403 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.138407 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.138409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.138497 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.139022 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:01.139031 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:01.139166 (+ 135us) client_negotiation.cc:769] Sending connection context
0811 04:41:01.139323 (+ 157us) client_negotiation.cc:240] Negotiation successful
0811 04:41:01.139531 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":146,"threads_started":1}
I20260811 04:41:01.140756 27349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:01.142560 27181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.131939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42130)
0811 04:41:01.132069 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.132084 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:01.132165 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:01.132530 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:01.132533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:01.132711 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:01.132800 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.132806 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.133710 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.133716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.134354 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.134371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.134926 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.134930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.135038 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.135469 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:01.135491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:01.136817 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:01.140813 (+ 3996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.140816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.140818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.141071 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.141429 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.141432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.141435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.141536 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.142065 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:01.142071 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:01.142163 (+ 92us) client_negotiation.cc:769] Sending connection context
0811 04:41:01.142276 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 04:41:01.142412 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260811 04:41:01.143067 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.132046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.2.238.188:35251)
0811 04:41:01.132335 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.132342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.132352 (+ 10us) server_negotiation.cc:407] Connection header received
0811 04:41:01.132378 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.132379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.132409 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.132477 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.132934 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.133622 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.134461 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.134787 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.134905 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.137161 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.137164 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.137167 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.137192 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.139610 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.141206 (+ 1596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.141208 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.141209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.141314 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.141683 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.141689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.141693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.141946 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.142746 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.142817 (+ 71us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.142911 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":109,"threads_started":1}
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:01.145556 27262 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } attempt: 3
I20260811 04:41:01.146461 27178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: Bootstrap starting.
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:41:01.149227 27178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:01.150170 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.140176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57800 (local address 127.2.238.190:41629)
0811 04:41:01.140295 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.140301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.140374 (+ 73us) server_negotiation.cc:407] Connection header received
0811 04:41:01.140466 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.140469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.140509 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.140578 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.141312 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.142016 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.142808 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.143008 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.143107 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.145831 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.145835 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.145840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.145864 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.147666 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.148221 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.148225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.148228 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.148319 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.148668 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.148673 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.148677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.148905 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.149748 (+ 843us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.149865 (+ 117us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.149987 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:41:01.150177 27178 log.cc:824] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: Log is configured to *not* fsync() on all Append() calls
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:01.152735 27178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: No bootstrap required, opened a new log
I20260811 04:41:01.153224 27266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.141548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.2.238.189:35615)
0811 04:41:01.141685 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.141690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.141713 (+ 23us) server_negotiation.cc:407] Connection header received
0811 04:41:01.141753 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.141756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.141818 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.141906 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.142351 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.143005 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.144528 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.144863 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.144963 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.147867 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.147873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.147878 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.147904 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.149910 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.150415 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.150419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.150420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.150509 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.150857 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.150860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.150862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.151085 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.151158 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.152893 (+ 1735us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.153078 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260811 04:41:01.153935 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.141274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:35615 (local address 127.0.0.1:47146)
0811 04:41:01.141542 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.141555 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:01.141638 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:01.141923 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:01.141926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:01.142089 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:01.142206 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.142213 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.143495 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.143499 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.144401 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.144413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.144987 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.144991 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.145087 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.145560 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:01.145576 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:01.147723 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:01.150047 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.150050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.150052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.150299 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.150647 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.150651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.150656 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.150749 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.151237 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:01.151254 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:01.151346 (+ 92us) client_negotiation.cc:769] Sending connection context
0811 04:41:01.153374 (+ 2028us) client_negotiation.cc:240] Negotiation successful
0811 04:41:01.153539 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260811 04:41:01.154654 27349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:01.155184 27268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.145927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42146)
0811 04:41:01.146176 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.146208 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:01.146281 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:01.146770 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:01.146773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:01.146943 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:01.147027 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.147033 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.148038 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.148042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.148741 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.148752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.149377 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.149381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.149467 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.149971 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:01.149985 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:01.151380 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:01.153219 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.153222 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.153224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.153439 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.153865 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.153868 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.153869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.153951 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.154633 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:01.154635 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:01.154742 (+ 107us) client_negotiation.cc:769] Sending connection context
0811 04:41:01.154843 (+ 101us) client_negotiation.cc:240] Negotiation successful
0811 04:41:01.154996 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260811 04:41:01.156024 27354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.146445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.2.238.188:35251)
0811 04:41:01.146550 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.146553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.146565 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:41:01.146595 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.146598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.146633 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.147135 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.147216 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.147906 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.148880 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.149250 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.149680 (+ 430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.151517 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.151520 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.151522 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.151544 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.153102 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.153624 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.153627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.153628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.153718 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.154053 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.154057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.154059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.154482 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.155709 (+ 1227us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.155787 (+ 78us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.155891 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:41:01.157859 27178 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.158157 27178 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f694b5cffa49459b2cdc3489e030bc, State: Initialized, Role: FOLLOWER
I20260811 04:41:01.158550 27178 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.159180 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: Bootstrap starting.
I20260811 04:41:01.159994 27178 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: configured and running, proceeding with master startup.
Aug 11 04:41:01 dist-test-slave-0kls krb5kdc[27085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423261, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:01.162011 27262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:01.162935 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.151351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.2.238.189:35615)
0811 04:41:01.151989 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.151999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.152014 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:41:01.152049 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.152053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.152114 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.152955 (+ 841us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.153076 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.153752 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.154827 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.155147 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.155246 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.158296 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.158300 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.158302 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.158326 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.160398 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.160913 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.160916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.160917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.161012 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.161370 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.161374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.161375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.161586 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.162493 (+ 907us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.162585 (+ 92us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.162716 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":89,"threads_started":1}
I20260811 04:41:01.162961 27262 log.cc:824] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: Log is configured to *not* fsync() on all Append() calls
I20260811 04:41:01.165220 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.155330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.2.238.188:35251)
0811 04:41:01.155424 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.155428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.155996 (+ 568us) server_negotiation.cc:407] Connection header received
0811 04:41:01.156144 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.156147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.156186 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.156232 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.156801 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.157611 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.158753 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.159150 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.159236 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.161486 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.161489 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.161492 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.161512 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.163227 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.163698 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.163701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.163703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.163783 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.164360 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.164374 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.164375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.164533 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.164609 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.164959 (+ 350us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.165086 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260811 04:41:01.165618 27353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.155265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:35251 (local address 127.0.0.1:42158)
0811 04:41:01.155932 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:01.155948 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:01.156051 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:01.156377 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:01.156383 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:01.156560 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:01.156675 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.156683 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.157774 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.157778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.158622 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:01.158635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.159393 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.159397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:01.159493 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.159945 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:01.159961 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:01.161370 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:01.163341 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.163343 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.163345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.163582 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.164183 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:01.164185 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:01.164187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:01.164274 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:01.164685 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:01.164688 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:01.164785 (+ 97us) client_negotiation.cc:769] Sending connection context
0811 04:41:01.165354 (+ 569us) client_negotiation.cc:240] Negotiation successful
0811 04:41:01.165476 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587}
I20260811 04:41:01.166630 27262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: No bootstrap required, opened a new log
I20260811 04:41:01.167160 27358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.167375 27358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: This master's current role is: FOLLOWER
W20260811 04:41:01.168131 27369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 04:41:01.164968 27099 master_runner.cc:186] Error getting master registration for 127.2.238.189:35615: OK, instance_id { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" instance_seqno: 1786423260888457 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:41:01.168411 27099 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 04:41:01.168512 27369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:41:01.168861 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:41:01.170240 27146 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d0f694b5cffa49459b2cdc3489e030bc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57800
I20260811 04:41:01.170457 27146 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 04:41:01.170379 27262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.170601 27262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 099f7c7e1259487bab991a8d42dbf4dd, State: Initialized, Role: FOLLOWER
I20260811 04:41:01.170606 27146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:01.170922 27262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.171286 27349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: Bootstrap starting.
I20260811 04:41:01.171398 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.171527 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.171805 27146 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.172184 27262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:41:01.172519 27146 leader_election.cc:288] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [CANDIDATE]: Term 1 election: Requested vote from peers 099f7c7e1259487bab991a8d42dbf4dd (127.2.238.189:35615), a2e6994c48bd4df08d2de3f7eaf1dd0b (127.2.238.188:35251)
I20260811 04:41:01.172991 27231 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0f694b5cffa49459b2cdc3489e030bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "099f7c7e1259487bab991a8d42dbf4dd"
I20260811 04:41:01.173205 27231 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:01.173859 27349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:01.174536 27231 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f694b5cffa49459b2cdc3489e030bc in term 1.
I20260811 04:41:01.174767 27349 log.cc:824] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: Log is configured to *not* fsync() on all Append() calls
I20260811 04:41:01.175259 27121 leader_election.cc:302] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 099f7c7e1259487bab991a8d42dbf4dd, d0f694b5cffa49459b2cdc3489e030bc; no voters:
I20260811 04:41:01.175530 27358 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:41:01.175741 27358 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [term 1 LEADER]: Becoming Leader. State: Replica: d0f694b5cffa49459b2cdc3489e030bc, State: Running, Role: LEADER
I20260811 04:41:01.176052 27358 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.181499 27390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f694b5cffa49459b2cdc3489e030bc. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.181741 27390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: This master's current role is: LEADER
W20260811 04:41:01.185007 27388 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260811 04:41:01.186036 27395 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:41:01.182399 27185 master_runner.cc:186] Error getting master registration for 127.2.238.188:35251: OK, instance_id { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" instance_seqno: 1786423261116937 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:41:01.186555 27185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260811 04:41:01.186614 27388 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:41:01.187108 27395 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:41:01.188578 27349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: No bootstrap required, opened a new log
I20260811 04:41:01.195034 27356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.179055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.2.238.189:35615)
0811 04:41:01.179152 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.179156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.185351 (+ 6195us) server_negotiation.cc:407] Connection header received
0811 04:41:01.185517 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.185520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.185567 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.185615 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.186466 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.187268 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.189147 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.189469 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.189545 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.191212 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.191215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.191218 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.191242 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.192940 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.193471 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.193475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.193477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.193563 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.193920 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.193925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.193928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.194171 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.194328 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.194745 (+ 417us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.194871 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 04:41:01.195019 27349 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.197154 27349 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e6994c48bd4df08d2de3f7eaf1dd0b, State: Initialized, Role: FOLLOWER
I20260811 04:41:01.197923 27349 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } }
I20260811 04:41:01.199862 27319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0f694b5cffa49459b2cdc3489e030bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b"
I20260811 04:41:01.200093 27319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:01.201359 27231 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [term 1 FOLLOWER]: Refusing update from remote peer d0f694b5cffa49459b2cdc3489e030bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:41:01.202013 27390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:41:01.202080 27319 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0f694b5cffa49459b2cdc3489e030bc in term 1.
I20260811 04:41:01.203037 27317 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [term 1 FOLLOWER]: Refusing update from remote peer d0f694b5cffa49459b2cdc3489e030bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:41:01.204972 27390 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:41:01.205696 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f694b5cffa49459b2cdc3489e030bc. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.205840 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.206069 27180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.179820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.2.238.190:41629)
0811 04:41:01.179963 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.179969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.195591 (+ 15622us) server_negotiation.cc:407] Connection header received
0811 04:41:01.195783 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.195789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.195843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.195896 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.198208 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.198881 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.200493 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.200808 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.200892 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.202505 (+ 1613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.202509 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.202511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.202535 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.204190 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.204682 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.204685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.204686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.204772 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.205080 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.205083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.205084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.205299 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.205404 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.205755 (+ 351us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.205927 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260811 04:41:01.209003 27352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:01.178341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42166 (local address 127.2.238.188:35251)
0811 04:41:01.178451 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.178455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.179618 (+ 1163us) server_negotiation.cc:407] Connection header received
0811 04:41:01.179868 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.179873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.179922 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.179978 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.180931 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.181625 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.183270 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:01.183561 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:01.183630 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:01.185494 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:01.185498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:01.185501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:01.185526 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:01.188054 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.197066 (+ 9012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.197069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.197071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.197181 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:01.197527 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:01.197530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:01.197532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:01.197747 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:01.197876 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:01.199963 (+ 2087us) server_negotiation.cc:299] Negotiation successful
0811 04:41:01.200078 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:41:01.209810 27349 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:41:01.212143 27358 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.212280 27358 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: This master's current role is: LEADER
I20260811 04:41:01.212522 27398 mvcc.cc:205] Tried to move back new op lower bound from 7317189677869944832 to 7317189677780688896. Current Snapshot: MvccSnapshot[applied={T|T < 7317189677869944832}]
I20260811 04:41:01.212771 27403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0f694b5cffa49459b2cdc3489e030bc. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.212903 27403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.212994 27403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.213078 27405 mvcc.cc:205] Tried to move back new op lower bound from 7317189677869944832 to 7317189677780688896. Current Snapshot: MvccSnapshot[applied={T|T < 7317189677869944832}]
I20260811 04:41:01.213400 27403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.213313 27399 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.213542 27399 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.221175 27390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.221299 27390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc [sys.catalog]: This master's current role is: LEADER
I20260811 04:41:01.222184 27400 mvcc.cc:205] Tried to move back new op lower bound from 7317189677869944832 to 7317189677780688896. Current Snapshot: MvccSnapshot[applied={T|T < 7317189677869944832}]
I20260811 04:41:01.222312 27372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0f694b5cffa49459b2cdc3489e030bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0f694b5cffa49459b2cdc3489e030bc" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 41629 } } peers { permanent_uuid: "099f7c7e1259487bab991a8d42dbf4dd" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 35615 } } peers { permanent_uuid: "a2e6994c48bd4df08d2de3f7eaf1dd0b" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 35251 } } }
I20260811 04:41:01.222389 27372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:01.222731 27395 catalog_manager.cc:1347] Generated new cluster ID: 91abbf91c3984622a0fadf09bb9bec44
I20260811 04:41:01.222790 27395 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:41:01.228744 27421 catalog_manager.cc:1259] Loaded cluster ID: 91abbf91c3984622a0fadf09bb9bec44
I20260811 04:41:01.228801 27421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: loading cluster ID for follower catalog manager: success
W20260811 04:41:01.229264 27421 catalog_manager.cc:1576] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260811 04:41:01.229341 27421 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:41:01.237629 27395 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:41:01.238797 27395 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:41:01.247920 27395 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0f694b5cffa49459b2cdc3489e030bc: Generated new TSK 0
I20260811 04:41:01.248704 27395 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:41:01.281560 27352 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.279015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42180 (local address 127.2.238.188:35251)
0811 04:41:01.279147 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.279151 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.280487 (+ 1336us) server_negotiation.cc:407] Connection header received
0811 04:41:01.280682 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.280686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.280749 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.280802 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.281421 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42180: BlockingRecv error: recv got EOF from 127.0.0.1:42180 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260811 04:41:01.283088 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.279830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47178 (local address 127.2.238.189:35615)
0811 04:41:01.279950 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.279954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.282100 (+ 2146us) server_negotiation.cc:407] Connection header received
0811 04:41:01.282323 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.282327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.282376 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.282426 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.282964 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47178: BlockingRecv error: recv got EOF from 127.0.0.1:47178 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260811 04:41:01.284999 27180 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.283675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57808 (local address 127.2.238.190:41629)
0811 04:41:01.283808 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.283812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.284138 (+ 326us) server_negotiation.cc:407] Connection header received
0811 04:41:01.284298 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.284301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.284347 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.284395 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.284866 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57808: BlockingRecv error: recv got EOF from 127.0.0.1:57808 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 04:41:01.289449 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.287908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.2.238.189:35615)
0811 04:41:01.288008 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.288012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.288456 (+ 444us) server_negotiation.cc:407] Connection header received
0811 04:41:01.288680 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.288683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.288782 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.288846 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.289349 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47180: BlockingRecv error: recv got EOF from 127.0.0.1:47180 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260811 04:41:01.294188 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.292673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.2.238.189:35615)
0811 04:41:01.292772 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.292776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.293270 (+ 494us) server_negotiation.cc:407] Connection header received
0811 04:41:01.293452 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.293455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.293502 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.293562 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.294106 (+ 544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47196: BlockingRecv error: recv got EOF from 127.0.0.1:47196 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260811 04:41:01.299379 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.298060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.2.238.189:35615)
0811 04:41:01.298143 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.298147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.298596 (+ 449us) server_negotiation.cc:407] Connection header received
0811 04:41:01.298772 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.298775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.298821 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.298866 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.299292 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47208: BlockingRecv error: recv got EOF from 127.0.0.1:47208 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260811 04:41:01.306653 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.305280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.2.238.189:35615)
0811 04:41:01.305399 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.305402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.305869 (+ 467us) server_negotiation.cc:407] Connection header received
0811 04:41:01.306039 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.306042 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.306083 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.306128 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.306573 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47212: BlockingRecv error: recv got EOF from 127.0.0.1:47212 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260811 04:41:01.317730 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.316337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.2.238.189:35615)
0811 04:41:01.316445 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.316449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.316925 (+ 476us) server_negotiation.cc:407] Connection header received
0811 04:41:01.317089 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.317092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.317133 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.317179 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.317629 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47224: BlockingRecv error: recv got EOF from 127.0.0.1:47224 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260811 04:41:01.337935 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.336090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47228 (local address 127.2.238.189:35615)
0811 04:41:01.336213 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.336218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.336857 (+ 639us) server_negotiation.cc:407] Connection header received
0811 04:41:01.337081 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.337084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.337141 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.337193 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.337821 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47228: BlockingRecv error: recv got EOF from 127.0.0.1:47228 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260811 04:41:01.374670 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.372760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47244 (local address 127.2.238.189:35615)
0811 04:41:01.372911 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.372916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.373584 (+ 668us) server_negotiation.cc:407] Connection header received
0811 04:41:01.373863 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.373867 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.373923 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.373975 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.374551 (+ 576us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47244: BlockingRecv error: recv got EOF from 127.0.0.1:47244 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260811 04:41:01.442526 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.441369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.2.238.189:35615)
0811 04:41:01.441549 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.441553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.441657 (+ 104us) server_negotiation.cc:407] Connection header received
0811 04:41:01.441821 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.441823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.441871 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.441945 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.442397 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47256: BlockingRecv error: recv got EOF from 127.0.0.1:47256 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 04:41:01.574314 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.572901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.2.238.189:35615)
0811 04:41:01.573057 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.573061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.573326 (+ 265us) server_negotiation.cc:407] Connection header received
0811 04:41:01.573433 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.573436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.573492 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.573566 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.574139 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47262: BlockingRecv error: recv got EOF from 127.0.0.1:47262 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 04:41:01.833721 27356 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:01.832635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.2.238.189:35615)
0811 04:41:01.832811 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:01.832815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:01.832833 (+ 18us) server_negotiation.cc:407] Connection header received
0811 04:41:01.832972 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:01.832974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:01.833029 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:01.833107 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:01.833574 (+ 467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47264: BlockingRecv error: recv got EOF from 127.0.0.1:47264 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260811 04:41:02.188040 27388 catalog_manager.cc:1259] Loaded cluster ID: 91abbf91c3984622a0fadf09bb9bec44
I20260811 04:41:02.188127 27388 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: loading cluster ID for follower catalog manager: success
I20260811 04:41:02.189816 27388 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: acquiring CA information for follower catalog manager: success
I20260811 04:41:02.190953 27388 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 099f7c7e1259487bab991a8d42dbf4dd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:41:02.231820 27421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: acquiring CA information for follower catalog manager: success
I20260811 04:41:02.233122 27421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a2e6994c48bd4df08d2de3f7eaf1dd0b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:41:02.351912 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:02.348231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47280 (local address 127.2.238.189:35615)
0811 04:41:02.348513 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:02.348517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:02.348553 (+ 36us) server_negotiation.cc:407] Connection header received
0811 04:41:02.348605 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:02.348607 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:02.348798 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:02.348889 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:02.349143 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.349662 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.350493 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.350711 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.350789 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:02.351077 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:02.351600 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:02.351689 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:02.351746 (+ 57us) server_negotiation.cc:299] Negotiation successful
0811 04:41:02.351792 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260811 04:41:02.358518 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:02.355048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.2.238.188:35251)
0811 04:41:02.355332 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:02.355335 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:02.355347 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:41:02.355418 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:02.355420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:02.355540 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:02.355625 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:02.355841 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.356352 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.357123 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.357348 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.357423 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:02.357674 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:02.358117 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:02.358218 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:02.358264 (+ 46us) server_negotiation.cc:299] Negotiation successful
0811 04:41:02.358307 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":83,"threads_started":1}
I20260811 04:41:02.365940 27507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:02.361953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.2.238.188:35251)
0811 04:41:02.362065 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:02.362069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:02.362295 (+ 226us) server_negotiation.cc:407] Connection header received
0811 04:41:02.362504 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:02.362510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:02.362562 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:02.362632 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:02.362862 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.363507 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.364492 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.364848 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.364935 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:02.365434 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:02.365495 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:02.365542 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:02.365730 (+ 188us) server_negotiation.cc:299] Negotiation successful
0811 04:41:02.365776 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260811 04:41:02.366096 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:02.361521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.2.238.189:35615)
0811 04:41:02.361870 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:02.361875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:02.362241 (+ 366us) server_negotiation.cc:407] Connection header received
0811 04:41:02.362401 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:02.362404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:02.362478 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:02.362544 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:02.362810 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.363486 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.364609 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.364867 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.364969 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:02.365391 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:02.365464 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:02.365502 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:02.365672 (+ 170us) server_negotiation.cc:299] Negotiation successful
0811 04:41:02.365740 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244}
I20260811 04:41:02.367280 27514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:02.361689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57818 (local address 127.2.238.190:41629)
0811 04:41:02.361970 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:02.361975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:02.362341 (+ 366us) server_negotiation.cc:407] Connection header received
0811 04:41:02.362811 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:02.362814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:02.362959 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:02.363075 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:02.363343 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.364108 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.365376 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:02.365690 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:02.365813 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:02.366264 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:02.366607 (+ 343us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:02.366673 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:02.367082 (+ 409us) server_negotiation.cc:299] Negotiation successful
0811 04:41:02.367163 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":110,"threads_started":1}
I20260811 04:41:02.370381 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27099
I20260811 04:41:02.380342 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27185
I20260811 04:41:02.388236 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27271
2026-08-11T04:41:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3106 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 11 04:41:02 dist-test-slave-0kls krb5kdc[27518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 11 04:41:02 dist-test-slave-0kls krb5kdc[27518](info): set up 2 sockets
Aug 11 04:41:02 dist-test-slave-0kls krb5kdc[27518](info): commencing operation
krb5kdc: starting...
W20260811 04:41:03.454581 3002 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/test-user.keytab.
Aug 11 04:41:03 dist-test-slave-0kls krb5kdc[27518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-11T04:41:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-11T04:41:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.238.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
Entry for principal kudu/127.2.238.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab.
I20260811 04:41:03.580426 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:44773
--webserver_interface=127.2.238.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260811 04:41:03.692394 27532 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:03.692641 27532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:41:03.692699 27532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:03.692762 27532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260811 04:41:03.696578 27532 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:03.696628 27532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:03.696647 27532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:03.696702 27532 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:03.696738 27532 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:03.701846 27532 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.190.keytab
--principal=kudu/127.2.238.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.190:44773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.238.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:03.702877 27532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:03.703890 27532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:03.712514 27538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:03.712553 27532 server_base.cc:1034] running on GCE node
W20260811 04:41:03.712553 27541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:03.712512 27539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:03.713352 27532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:03.714527 27532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:03.715688 27532 hybrid_clock.cc:648] HybridClock initialized: now 1786423263715660 us; error 52 us; skew 500 ppm
Aug 11 04:41:03 dist-test-slave-0kls krb5kdc[27518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:03.722078 27532 init.cc:377] Logged in from keytab as kudu/127.2.238.190@KRBTEST.COM (short username kudu)
I20260811 04:41:03.723076 27532 webserver.cc:460] Webserver started at http://127.2.238.190:32903/ using document root <none> and password file <none>
I20260811 04:41:03.723655 27532 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:03.723748 27532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:03.723966 27532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:03.726207 27532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data/instance:
uuid: "03871b7eb55642c38858200c4b3ba98f"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "94cf4ae104842e62b22a59bef13c9a2c"
server_key_iv: "100962b3789e75f548caf6a62064aa46"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.726821 27532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance:
uuid: "03871b7eb55642c38858200c4b3ba98f"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "94cf4ae104842e62b22a59bef13c9a2c"
server_key_iv: "100962b3789e75f548caf6a62064aa46"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.730623 27532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260811 04:41:03.732916 27548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:03.734243 27532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:41:03.734375 27532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal
uuid: "03871b7eb55642c38858200c4b3ba98f"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "94cf4ae104842e62b22a59bef13c9a2c"
server_key_iv: "100962b3789e75f548caf6a62064aa46"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.734488 27532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:03.758942 27532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:03.762010 27532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:03.762254 27532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:03.769531 27532 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.190:44773
I20260811 04:41:03.769881 27608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.190:44773 every 8 connection(s)
I20260811 04:41:03.770552 27532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/data/info.pb
I20260811 04:41:03.773343 27609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:03.777107 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27532
I20260811 04:41:03.777256 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-0/wal/instance
I20260811 04:41:03.777063 27609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:03.777580 3002 external_mini_cluster.cc:1428] Setting key bee560cb2eae044898007394db16b006
WARNING: no policy specified for kudu/127.2.238.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.189@KRBTEST.COM" created.
Aug 11 04:41:03 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:03.796415 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.779601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.2.238.190:44773)
0811 04:41:03.780211 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:03.780236 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:03.780286 (+ 50us) server_negotiation.cc:407] Connection header received
0811 04:41:03.781336 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:03.781345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:03.781605 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:03.781822 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:03.782595 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:03.784080 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:03.785004 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:03.785590 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:03.785802 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:03.791066 (+ 5264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:03.791076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:03.791096 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:03.791118 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:03.793236 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:03.793866 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:03.793871 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:03.793872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:03.793972 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:03.794358 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:03.794360 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:03.794361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:03.794811 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:03.794915 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:03.795403 (+ 488us) server_negotiation.cc:299] Negotiation successful
0811 04:41:03.795601 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":478,"spinlock_wait_cycles":14080,"thread_start_us":200,"threads_started":1}
I20260811 04:41:03.796417 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.779600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:44773 (local address 127.0.0.1:36864)
0811 04:41:03.780103 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.780155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:03.780972 (+ 817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:03.781869 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:03.781873 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:03.782185 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:03.782437 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:03.782449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:03.784199 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:03.784201 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:03.784887 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:03.784895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:03.785763 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:03.785765 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:03.785932 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:03.786479 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:03.786495 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:03.790899 (+ 4404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:03.793347 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:03.793359 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:03.793372 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:03.793702 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:03.794064 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:03.794070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:03.794074 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:03.794258 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:03.794938 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:03.794945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:03.795125 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 04:41:03.795330 (+ 205us) client_negotiation.cc:240] Negotiation successful
0811 04:41:03.795699 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":61,"thread_start_us":162,"threads_started":1}
I20260811 04:41:03.798961 27609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:03.799669 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.799446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45832)
0811 04:41:03.799555 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.799621 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:03.799878 27552 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
I20260811 04:41:03.800721 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.800531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45840)
0811 04:41:03.800623 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.800665 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260811 04:41:03.800910 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:45423: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
Entry for principal kudu/127.2.238.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab.
I20260811 04:41:03.808274 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:45423
--webserver_interface=127.2.238.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:41:03.850185 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } attempt: 1
I20260811 04:41:03.851073 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.850772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45854)
0811 04:41:03.850934 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.850995 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260811 04:41:03.851830 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.851587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45868)
0811 04:41:03.851695 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.851754 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260811 04:41:03.851962 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:45423: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
W20260811 04:41:03.916675 27617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:03.916971 27617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260811 04:41:03.917040 27617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:03.917122 27617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 04:41:03.920110 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } attempt: 2
I20260811 04:41:03.920924 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.920614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45872)
0811 04:41:03.920779 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.920848 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260811 04:41:03.921238 27617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:03.921300 27617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:03.921334 27617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:03.921363 27617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:03.921398 27617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:03.921733 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:03.921529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45888)
0811 04:41:03.921633 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:03.921684 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260811 04:41:03.921922 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:45423: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
I20260811 04:41:03.926627 27617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.189.keytab
--principal=kudu/127.2.238.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.189:45423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.238.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:03.927618 27617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:03.928618 27617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:03.936872 27622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:03.936856 27625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:03.936848 27623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:03.937260 27617 server_base.cc:1034] running on GCE node
I20260811 04:41:03.937848 27617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:03.938912 27617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:03.940069 27617 hybrid_clock.cc:648] HybridClock initialized: now 1786423263940045 us; error 50 us; skew 500 ppm
Aug 11 04:41:03 dist-test-slave-0kls krb5kdc[27518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:03.942816 27617 init.cc:377] Logged in from keytab as kudu/127.2.238.189@KRBTEST.COM (short username kudu)
I20260811 04:41:03.943694 27617 webserver.cc:460] Webserver started at http://127.2.238.189:35685/ using document root <none> and password file <none>
I20260811 04:41:03.944245 27617 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:03.944339 27617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:03.944566 27617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:03.946609 27617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data/instance:
uuid: "0bf415ee0dda496caa4497eba79a170d"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "4b8ab103d0506d82cd9d6781e75983b1"
server_key_iv: "e1bf353c95d2ad18903c471b482503f4"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.947175 27617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal/instance:
uuid: "0bf415ee0dda496caa4497eba79a170d"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "4b8ab103d0506d82cd9d6781e75983b1"
server_key_iv: "e1bf353c95d2ad18903c471b482503f4"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.950630 27617 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260811 04:41:03.952934 27633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:03.954207 27617 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260811 04:41:03.954348 27617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal
uuid: "0bf415ee0dda496caa4497eba79a170d"
format_stamp: "Formatted at 2026-08-11 04:41:03 on dist-test-slave-0kls"
server_key: "4b8ab103d0506d82cd9d6781e75983b1"
server_key_iv: "e1bf353c95d2ad18903c471b482503f4"
server_key_version: "encryptionkey@0"
I20260811 04:41:03.954455 27617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:04.008522 27617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:04.011525 27617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:04.011756 27617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:04.013116 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } attempt: 3
I20260811 04:41:04.014055 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.013688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45890)
0811 04:41:04.013878 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.013952 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260811 04:41:04.014755 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.014509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45906)
0811 04:41:04.014625 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.014683 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:04.014894 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.189:45423: Network error: Client connection negotiation failed: client connection to 127.2.238.189:45423: connect: Connection refused (error 111)
I20260811 04:41:04.019335 27617 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.189:45423
I20260811 04:41:04.019611 27690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.189:45423 every 8 connection(s)
I20260811 04:41:04.020318 27617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/data/info.pb
I20260811 04:41:04.023175 27691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:04.024610 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27617
I20260811 04:41:04.024704 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-1/wal/instance
I20260811 04:41:04.025010 3002 external_mini_cluster.cc:1428] Setting key 61a09b29fa7a47a8e7b74dabcd73a99b
I20260811 04:41:04.027186 27691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.238.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.238.188@KRBTEST.COM" created.
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:04.042611 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.029375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36872 (local address 127.2.238.190:44773)
0811 04:41:04.029513 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.029517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.030591 (+ 1074us) server_negotiation.cc:407] Connection header received
0811 04:41:04.031520 (+ 929us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.031523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.031584 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.031684 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.032820 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.033349 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.034609 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.034839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.034978 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.037855 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.037858 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.037860 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.037885 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.040117 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.040673 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.040677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.040679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.040773 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.041131 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.041137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.041142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.041440 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.041558 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.042186 (+ 628us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.042358 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:41:04.043224 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.029949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:44773 (local address 127.0.0.1:36872)
0811 04:41:04.030447 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.030489 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.031291 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.031859 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.031866 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.032208 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.032670 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.032682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.033484 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.033487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.034440 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.034450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.035039 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.035044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.035216 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.035889 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.035906 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.037604 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.040259 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.040265 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.040280 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.040557 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.040871 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.040875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.040877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.041029 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.041580 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.041585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.041903 (+ 318us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.042073 (+ 170us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.042332 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":136,"threads_started":1}
I20260811 04:41:04.044701 27691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
Entry for principal kudu/127.2.238.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab.
I20260811 04:41:04.053958 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.045221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45922)
0811 04:41:04.045348 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.045363 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.045441 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.046120 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.046123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.046262 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.046350 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.046355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.047459 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.047466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.047976 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.047984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.048572 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.048575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.048669 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.049187 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.049198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.050482 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.052442 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.052444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.052446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.052654 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.052959 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.052962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.052964 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.053045 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.053445 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.053447 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.053520 (+ 73us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.053598 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.053710 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260811 04:41:04.054251 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.045328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45922 (local address 127.2.238.189:45423)
0811 04:41:04.045663 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.045676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.045695 (+ 19us) server_negotiation.cc:407] Connection header received
0811 04:41:04.045742 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.045746 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.046000 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.046111 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.046499 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.047341 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.048095 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.048454 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.048572 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.050610 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.050618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.050628 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.050650 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.052333 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.052752 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.052757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.052758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.052833 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.053129 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.053131 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.053132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.053355 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.053454 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.053787 (+ 333us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.053978 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":165,"threads_started":1}
I20260811 04:41:04.054916 3002 external_mini_cluster.cc:1326] Running /tmp/dist-test-taski3hQIv/build/debug/bin/kudu
/tmp/dist-test-taski3hQIv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:39937
--webserver_interface=127.2.238.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.238.148:44919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260811 04:41:04.056062 27691 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:04.056823 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.056513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35224)
0811 04:41:04.056640 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.056744 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260811 04:41:04.057055 27637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
I20260811 04:41:04.057971 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.057660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35240)
0811 04:41:04.057836 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.057909 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260811 04:41:04.058118 27691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:39937: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
I20260811 04:41:04.107414 27691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } attempt: 1
I20260811 04:41:04.108389 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.108080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35250)
0811 04:41:04.108243 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.108308 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260811 04:41:04.109180 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.108925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35266)
0811 04:41:04.109059 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.109113 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260811 04:41:04.109360 27691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:39937: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
I20260811 04:41:04.158121 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } attempt: 4
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
I20260811 04:41:04.168277 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.158752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45928)
0811 04:41:04.158969 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.158989 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.159107 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.159412 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.159414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.159678 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.159801 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.159809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.161526 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.161529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.162175 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.162185 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.162801 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.162805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.162928 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.163538 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.163550 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.164964 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.166897 (+ 1933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.166900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.166901 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.167101 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.167379 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.167381 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.167382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.167457 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.167847 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.167849 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.167930 (+ 81us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.168008 (+ 78us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.168130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260811 04:41:04.168381 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.158816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45928 (local address 127.2.238.189:45423)
0811 04:41:04.158979 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.158984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.159052 (+ 68us) server_negotiation.cc:407] Connection header received
0811 04:41:04.159223 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.159227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.159287 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.159378 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.159944 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.161385 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.162304 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.162655 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.162783 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.165104 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.165109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.165112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.165150 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.166775 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.167196 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.167200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.167202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.167286 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.167541 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.167544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.167546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.167761 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.167850 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.168096 (+ 246us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.168229 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260811 04:41:04.169198 27609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
W20260811 04:41:04.169582 27697 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260811 04:41:04.169977 27697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260811 04:41:04.169992 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.169679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35278)
0811 04:41:04.169854 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.169900 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260811 04:41:04.170023 27697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260811 04:41:04.170079 27697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260811 04:41:04.170680 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.170501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35284)
0811 04:41:04.170591 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.170634 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260811 04:41:04.170820 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:39937: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
W20260811 04:41:04.174465 27697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260811 04:41:04.174516 27697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260811 04:41:04.174537 27697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260811 04:41:04.174594 27697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260811 04:41:04.174636 27697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260811 04:41:04.177547 27691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } attempt: 2
I20260811 04:41:04.178210 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.177963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35292)
0811 04:41:04.178094 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.178146 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260811 04:41:04.178956 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.178748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35304)
0811 04:41:04.178865 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.178908 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260811 04:41:04.179152 27691 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:39937: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
I20260811 04:41:04.180456 27697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.238.148:44919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.238.190:44773,127.2.238.189:45423,127.2.238.188:39937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/krb5kdc/kudu_127.2.238.188.keytab
--principal=kudu/127.2.238.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.238.188:39937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.238.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 11 Aug 2026 03:56:35 UTC on ccycloud.quasar-iyycfb.root.comops.site
build id 4579566
I20260811 04:41:04.181509 27697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260811 04:41:04.182600 27697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260811 04:41:04.190948 27707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:04.190958 27704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260811 04:41:04.190964 27705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260811 04:41:04.191030 27697 server_base.cc:1034] running on GCE node
I20260811 04:41:04.191831 27697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260811 04:41:04.192932 27697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260811 04:41:04.194239 27697 hybrid_clock.cc:648] HybridClock initialized: now 1786423264194200 us; error 49 us; skew 500 ppm
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260811 04:41:04.196760 27697 init.cc:377] Logged in from keytab as kudu/127.2.238.188@KRBTEST.COM (short username kudu)
I20260811 04:41:04.197639 27697 webserver.cc:460] Webserver started at http://127.2.238.188:46857/ using document root <none> and password file <none>
I20260811 04:41:04.198263 27697 fs_manager.cc:359] Metadata directory not provided
I20260811 04:41:04.198354 27697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260811 04:41:04.198580 27697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260811 04:41:04.200562 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data/instance:
uuid: "fae13dc630c341a78db74e8b1c5dbaed"
format_stamp: "Formatted at 2026-08-11 04:41:04 on dist-test-slave-0kls"
server_key: "8797ef3c976a5204632f5669d2365d91"
server_key_iv: "863f1a9222af4a537ea5e57e363283a6"
server_key_version: "encryptionkey@0"
I20260811 04:41:04.201134 27697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal/instance:
uuid: "fae13dc630c341a78db74e8b1c5dbaed"
format_stamp: "Formatted at 2026-08-11 04:41:04 on dist-test-slave-0kls"
server_key: "8797ef3c976a5204632f5669d2365d91"
server_key_iv: "863f1a9222af4a537ea5e57e363283a6"
server_key_version: "encryptionkey@0"
I20260811 04:41:04.204771 27697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260811 04:41:04.207122 27716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260811 04:41:04.208307 27697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260811 04:41:04.208446 27697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data,/tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal
uuid: "fae13dc630c341a78db74e8b1c5dbaed"
format_stamp: "Formatted at 2026-08-11 04:41:04 on dist-test-slave-0kls"
server_key: "8797ef3c976a5204632f5669d2365d91"
server_key_iv: "863f1a9222af4a537ea5e57e363283a6"
server_key_version: "encryptionkey@0"
I20260811 04:41:04.208550 27697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260811 04:41:04.227233 27697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260811 04:41:04.229916 27697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260811 04:41:04.229980 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } attempt: 1
I20260811 04:41:04.230098 27697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260811 04:41:04.230733 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.230483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35312)
0811 04:41:04.230609 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.230660 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260811 04:41:04.231520 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.231283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35318)
0811 04:41:04.231393 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.231449 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260811 04:41:04.231663 27609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.238.188:39937: Network error: Client connection negotiation failed: client connection to 127.2.238.188:39937: connect: Connection refused (error 111)
I20260811 04:41:04.237159 27697 rpc_server.cc:307] RPC server started. Bound to: 127.2.238.188:39937
I20260811 04:41:04.237439 27772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.238.188:39937 every 8 connection(s)
I20260811 04:41:04.238224 27697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/data/info.pb
I20260811 04:41:04.241040 27773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260811 04:41:04.242151 3002 external_mini_cluster.cc:1388] Started /tmp/dist-test-taski3hQIv/build/debug/bin/kudu as pid 27697
I20260811 04:41:04.242308 3002 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taski3hQIv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786422982503306-3002-0/minicluster-data/master-2/wal/instance
I20260811 04:41:04.242690 3002 external_mini_cluster.cc:1428] Setting key adbdc516bd40782e49057c43f81c77bb
I20260811 04:41:04.245476 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:04.256120 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.247118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36884 (local address 127.2.238.190:44773)
0811 04:41:04.247236 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.247240 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.247522 (+ 282us) server_negotiation.cc:407] Connection header received
0811 04:41:04.247636 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.247640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.247698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.247788 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.248502 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.249121 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.250155 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.250368 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.250472 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.252819 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.252822 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.252824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.252848 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.254225 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.254700 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.254704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.254706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.254792 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.255109 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.255113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.255114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.255303 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.255429 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.255797 (+ 368us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.255925 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260811 04:41:04.257035 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.244032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.2.238.188:39937)
0811 04:41:04.244465 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.244482 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.244516 (+ 34us) server_negotiation.cc:407] Connection header received
0811 04:41:04.245124 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.245139 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.245460 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.245812 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.246606 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.247947 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.248947 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.249524 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.249671 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.252238 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.252242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.252270 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.252296 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.254227 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.254673 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.254684 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.254685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.254787 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.255065 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.255069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.255070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.255400 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.255636 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.255820 (+ 184us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.256277 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":300,"thread_start_us":144,"threads_started":1}
I20260811 04:41:04.257509 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.247001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.190:44773 (local address 127.0.0.1:36884)
0811 04:41:04.247385 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.247432 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.247545 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.247858 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.247864 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.248264 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.248375 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.248381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.249239 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.249242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.250035 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.250043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.250522 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.250526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.250618 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.251095 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.251111 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.252664 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.254348 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.254359 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.254368 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.254594 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.254886 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.254890 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.254892 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.255009 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.255430 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.255437 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.255576 (+ 139us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.255744 (+ 168us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.257388 (+ 1644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":1370,"thread_start_us":97,"threads_started":1}
I20260811 04:41:04.259038 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } has no permanent_uuid. Determining permanent_uuid...
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.190@KRBTEST.COM
I20260811 04:41:04.267767 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.259537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.189:45423 (local address 127.0.0.1:45936)
0811 04:41:04.259712 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.259725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.259817 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.260097 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.260101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.260264 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.260334 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.260339 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.261078 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.261084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.261594 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.261602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.262119 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.262123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.262204 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.262682 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.262708 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.264081 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.266220 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.266222 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.266224 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.266418 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.266728 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.266735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.266739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.266834 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.267291 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.267293 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.267388 (+ 95us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.267469 (+ 81us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.267595 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260811 04:41:04.267859 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.259611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45936 (local address 127.2.238.189:45423)
0811 04:41:04.259723 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.259728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.259767 (+ 39us) server_negotiation.cc:407] Connection header received
0811 04:41:04.259942 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.259945 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.260000 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.260085 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.260448 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.260949 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.261707 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.262003 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.262108 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.264236 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.264239 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.264242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.264273 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.266102 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.266523 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.266530 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.266533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.266625 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.266938 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.266941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.266943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.267185 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.267295 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.267552 (+ 257us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.267675 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260811 04:41:04.268462 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.259639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36886 (local address 127.2.238.190:44773)
0811 04:41:04.259736 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.259740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.259887 (+ 147us) server_negotiation.cc:407] Connection header received
0811 04:41:04.260021 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.260024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.260065 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.260153 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.260738 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.261243 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.262015 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.262234 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.262337 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.264866 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.264871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.264877 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.264910 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.266455 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.266939 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.266942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.266944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.267025 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.267293 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.267296 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.267297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.267514 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.267755 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.268238 (+ 483us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.268351 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260811 04:41:04.268978 27773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } has no permanent_uuid. Determining permanent_uuid...
I20260811 04:41:04.270308 27691 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } attempt: 3
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423264, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.188@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.238.189@KRBTEST.COM
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.189@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:04.278434 27778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.269480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35344)
0811 04:41:04.269589 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.269601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.269667 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.270086 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.270089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.270244 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.270325 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.270329 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.271169 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.271171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.271928 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.271938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.272555 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.272559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.272637 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.273148 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.273164 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.274456 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.276441 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.276444 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.276446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.277011 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.277317 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.277322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.277327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.277408 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.277886 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.277890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.278008 (+ 118us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.278121 (+ 113us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.278264 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260811 04:41:04.278599 27776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.269630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.2.238.188:39937)
0811 04:41:04.269920 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.269923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.269933 (+ 10us) server_negotiation.cc:407] Connection header received
0811 04:41:04.269959 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.269961 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.269991 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.270075 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.270429 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.271088 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.272048 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.272441 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.272545 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.274589 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.274606 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.274610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.274640 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.276324 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.277115 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.277120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.277121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.277209 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.277495 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.277498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.277499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.277716 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.277893 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.278199 (+ 306us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.278371 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":209}
I20260811 04:41:04.279603 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.269724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.2.238.189:45423)
0811 04:41:04.269855 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.269859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.269871 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:41:04.269958 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.269961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.269996 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.270084 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.270799 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.271488 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.272422 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.272761 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.272874 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.275710 (+ 2836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.275714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.275717 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.275747 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.277530 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.278057 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.278060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.278062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.278146 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.278449 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.278455 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.278458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.278692 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.278940 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.279299 (+ 359us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.279464 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 04:41:04.280791 27694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.270661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35348)
0811 04:41:04.271046 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.271059 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.271135 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.271396 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.271398 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.271559 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.271632 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.271637 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.273006 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.273009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.273931 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.273946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.274676 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.274680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.274781 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.275361 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.275379 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.276780 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.279142 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.279148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.279153 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.279499 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.279828 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.279831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.279833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.279927 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.280400 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.280403 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.280502 (+ 99us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.280585 (+ 83us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.280681 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311}
I20260811 04:41:04.280946 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.270738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.2.238.188:39937)
0811 04:41:04.271086 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.271093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.271107 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:41:04.271255 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.271257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.271290 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.271380 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.271783 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.272867 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.274107 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.274554 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.274654 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.276919 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.276925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.276931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.276960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.279033 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.279613 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.279618 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.279620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.279714 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.280032 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.280035 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.280037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.280291 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.280394 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.280665 (+ 271us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.280775 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":164,"threads_started":1}
I20260811 04:41:04.282529 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: Bootstrap starting.
I20260811 04:41:04.283627 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: Bootstrap starting.
I20260811 04:41:04.284899 27773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:04.285624 27691 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:04.285636 27773 log.cc:824] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: Log is configured to *not* fsync() on all Append() calls
I20260811 04:41:04.286413 27691 log.cc:824] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: Log is configured to *not* fsync() on all Append() calls
I20260811 04:41:04.287950 27773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: No bootstrap required, opened a new log
I20260811 04:41:04.288528 27691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: No bootstrap required, opened a new log
I20260811 04:41:04.289160 3002 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260811 04:41:04.290552 27581 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "03871b7eb55642c38858200c4b3ba98f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36886
I20260811 04:41:04.291886 27691 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.291883 27773 raft_consensus.cc:348] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.292140 27773 raft_consensus.cc:729] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae13dc630c341a78db74e8b1c5dbaed, State: Initialized, Role: FOLLOWER
I20260811 04:41:04.292141 27691 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bf415ee0dda496caa4497eba79a170d, State: Initialized, Role: FOLLOWER
I20260811 04:41:04.292429 27691 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.292449 27773 consensus_queue.cc:260] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.292907 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.293040 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.293641 27691 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:41:04.293689 27773 sys_catalog.cc:564] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:41:04.295846 27783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.296096 27783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.298807 27609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } attempt: 2
W20260811 04:41:04.298868 27805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:41:04.298949 27805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260811 04:41:04.299749 27806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:41:04.299918 27806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:41:04.297698 27697 master_runner.cc:186] Error getting master registration for 127.2.238.190:44773: OK, instance_id { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" instance_seqno: 1786423263767465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:41:04.300973 27697 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260811 04:41:04.301221 27617 master_runner.cc:186] Error getting master registration for 127.2.238.190:44773: OK, instance_id { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" instance_seqno: 1786423263767465 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260811 04:41:04.304744 27617 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 11 04:41:04 dist-test-slave-0kls krb5kdc[27518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786423263, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.238.190@KRBTEST.COM for kudu/127.2.238.188@KRBTEST.COM
I20260811 04:41:04.310930 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.299261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.2.238.188:39937)
0811 04:41:04.299376 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.299380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.299395 (+ 15us) server_negotiation.cc:407] Connection header received
0811 04:41:04.299486 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.299489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.299537 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.299612 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.300292 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.301044 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.302278 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.302658 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.302756 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.305402 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.305406 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.305409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.305435 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.307371 (+ 1936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.307951 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.307954 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.307956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.308123 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.308705 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.308709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.308710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.309154 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.309639 (+ 485us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.309965 (+ 326us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.310361 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260811 04:41:04.310855 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.299161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.238.188:39937 (local address 127.0.0.1:35354)
0811 04:41:04.299315 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0811 04:41:04.299329 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0811 04:41:04.299398 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0811 04:41:04.299749 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0811 04:41:04.299752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0811 04:41:04.299993 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0811 04:41:04.300104 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.300112 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.301203 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.301230 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.302111 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0811 04:41:04.302123 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.302914 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.302918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0811 04:41:04.303005 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.303628 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0811 04:41:04.303644 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0811 04:41:04.305206 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0811 04:41:04.307520 (+ 2314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.307539 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.307541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.307808 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.308373 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0811 04:41:04.308378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0811 04:41:04.308382 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0811 04:41:04.308571 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0811 04:41:04.309360 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0811 04:41:04.309364 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0811 04:41:04.309544 (+ 180us) client_negotiation.cc:769] Sending connection context
0811 04:41:04.309881 (+ 337us) client_negotiation.cc:240] Negotiation successful
0811 04:41:04.310408 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260811 04:41:04.314495 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: Bootstrap starting.
I20260811 04:41:04.317406 27609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: Neither blocks nor log segments found. Creating new log.
I20260811 04:41:04.318384 27609 log.cc:824] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: Log is configured to *not* fsync() on all Append() calls
I20260811 04:41:04.321308 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: No bootstrap required, opened a new log
I20260811 04:41:04.325629 27609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.326021 27609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03871b7eb55642c38858200c4b3ba98f, State: Initialized, Role: FOLLOWER
I20260811 04:41:04.326527 27609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.327210 27811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.327409 27811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.328613 27581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260811 04:41:04.328735 27581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:04.328543 27609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: configured and running, proceeding with master startup.
I20260811 04:41:04.329957 27581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.330873 27581 leader_election.cc:288] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [CANDIDATE]: Term 1 election: Requested vote from peers 0bf415ee0dda496caa4497eba79a170d (127.2.238.189:45423), fae13dc630c341a78db74e8b1c5dbaed (127.2.238.188:39937)
I20260811 04:41:04.332149 27662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03871b7eb55642c38858200c4b3ba98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0bf415ee0dda496caa4497eba79a170d"
I20260811 04:41:04.332405 27662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:04.333709 27662 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03871b7eb55642c38858200c4b3ba98f in term 1.
I20260811 04:41:04.335868 27745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03871b7eb55642c38858200c4b3ba98f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fae13dc630c341a78db74e8b1c5dbaed"
I20260811 04:41:04.336097 27745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [term 0 FOLLOWER]: Advancing to term 1
I20260811 04:41:04.337930 27745 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03871b7eb55642c38858200c4b3ba98f in term 1.
I20260811 04:41:04.339128 27551 leader_election.cc:302] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03871b7eb55642c38858200c4b3ba98f, fae13dc630c341a78db74e8b1c5dbaed; no voters:
I20260811 04:41:04.339408 27811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 1 FOLLOWER]: Leader election won for term 1
I20260811 04:41:04.339586 27811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [term 1 LEADER]: Becoming Leader. State: Replica: 03871b7eb55642c38858200c4b3ba98f, State: Running, Role: LEADER
I20260811 04:41:04.339959 27811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } }
I20260811 04:41:04.341662 27811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03871b7eb55642c38858200c4b3ba98f. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.341761 27811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: This master's current role is: LEADER
W20260811 04:41:04.335652 27823 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260811 04:41:04.342655 27823 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260811 04:41:04.350350 27831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260811 04:41:04.351311 27831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260811 04:41:04.372072 27779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.343098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35366 (local address 127.2.238.188:39937)
0811 04:41:04.343207 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.343211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.350363 (+ 7152us) server_negotiation.cc:407] Connection header received
0811 04:41:04.350519 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.350522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.350568 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.350616 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.352642 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.353462 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.356675 (+ 3213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.357012 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.357083 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.364280 (+ 7197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.364283 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.364287 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.364314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.365916 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.368643 (+ 2727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.368647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.368649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.368737 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.369428 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.369431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.369433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.369639 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.369759 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.371779 (+ 2020us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.371913 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260811 04:41:04.373148 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.344036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36898 (local address 127.2.238.190:44773)
0811 04:41:04.344211 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.344216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.350071 (+ 5855us) server_negotiation.cc:407] Connection header received
0811 04:41:04.351628 (+ 1557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.351631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.351676 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.351723 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.353593 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.354342 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.357234 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.357560 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.357628 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.366436 (+ 8808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.366439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.366442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.366470 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.368129 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.369022 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.369025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.369027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.369104 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.370075 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.370079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.370080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.370296 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.370387 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.372894 (+ 2507us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.373007 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260811 04:41:04.374842 27745 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [term 1 FOLLOWER]: Refusing update from remote peer 03871b7eb55642c38858200c4b3ba98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:41:04.375561 27811 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:41:04.378470 27696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:04.345268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45946 (local address 127.2.238.189:45423)
0811 04:41:04.345384 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.345388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.356598 (+ 11210us) server_negotiation.cc:407] Connection header received
0811 04:41:04.359140 (+ 2542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.359144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.359195 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.359257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.362420 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.363139 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.370683 (+ 7544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:04.371034 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:04.371107 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:04.374982 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0811 04:41:04.374986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0811 04:41:04.374988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0811 04:41:04.375016 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0811 04:41:04.376650 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.377134 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.377138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.377140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.377235 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0811 04:41:04.377526 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0811 04:41:04.377530 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0811 04:41:04.377532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0811 04:41:04.377755 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0811 04:41:04.377876 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:04.378189 (+ 313us) server_negotiation.cc:299] Negotiation successful
0811 04:41:04.378311 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260811 04:41:04.379559 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03871b7eb55642c38858200c4b3ba98f. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.379670 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.380650 27662 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [term 1 FOLLOWER]: Refusing update from remote peer 03871b7eb55642c38858200c4b3ba98f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260811 04:41:04.381287 27829 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260811 04:41:04.385769 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.386190 27782 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.386325 27782 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.386343 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: This master's current role is: LEADER
I20260811 04:41:04.386027 27836 mvcc.cc:205] Tried to move back new op lower bound from 7317189690822701056 to 7317189690739781632. Current Snapshot: MvccSnapshot[applied={T|T < 7317189690822701056}]
I20260811 04:41:04.386795 27837 mvcc.cc:205] Tried to move back new op lower bound from 7317189690822701056 to 7317189690739781632. Current Snapshot: MvccSnapshot[applied={T|T < 7317189690822701056}]
I20260811 04:41:04.387737 27783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03871b7eb55642c38858200c4b3ba98f. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.387846 27783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.393260 27829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.393383 27829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f [sys.catalog]: This master's current role is: LEADER
I20260811 04:41:04.395176 27831 catalog_manager.cc:1347] Generated new cluster ID: 8ffc1788b4da4dea97d9a157635027a6
I20260811 04:41:04.395238 27831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260811 04:41:04.400446 27839 mvcc.cc:205] Tried to move back new op lower bound from 7317189690822701056 to 7317189690739781632. Current Snapshot: MvccSnapshot[applied={T|T < 7317189690822701056}]
I20260811 04:41:04.403223 27783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03871b7eb55642c38858200c4b3ba98f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03871b7eb55642c38858200c4b3ba98f" member_type: VOTER last_known_addr { host: "127.2.238.190" port: 44773 } } peers { permanent_uuid: "0bf415ee0dda496caa4497eba79a170d" member_type: VOTER last_known_addr { host: "127.2.238.189" port: 45423 } } peers { permanent_uuid: "fae13dc630c341a78db74e8b1c5dbaed" member_type: VOTER last_known_addr { host: "127.2.238.188" port: 39937 } } }
I20260811 04:41:04.403360 27783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d [sys.catalog]: This master's current role is: FOLLOWER
I20260811 04:41:04.423151 27831 catalog_manager.cc:1370] Generated new certificate authority record
I20260811 04:41:04.424487 27831 catalog_manager.cc:1504] Loading token signing keys...
I20260811 04:41:04.448966 27831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03871b7eb55642c38858200c4b3ba98f: Generated new TSK 0
I20260811 04:41:04.449905 27831 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260811 04:41:04.480533 27614 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.476974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36900 (local address 127.2.238.190:44773)
0811 04:41:04.477107 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.477112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.479208 (+ 2096us) server_negotiation.cc:407] Connection header received
0811 04:41:04.479613 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.479617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.479673 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.479737 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.480391 (+ 654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36900: BlockingRecv error: recv got EOF from 127.0.0.1:36900 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260811 04:41:04.481941 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.477999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45956 (local address 127.2.238.189:45423)
0811 04:41:04.478109 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.478113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.479526 (+ 1413us) server_negotiation.cc:407] Connection header received
0811 04:41:04.481094 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.481098 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.481152 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.481205 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.481779 (+ 574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45956: BlockingRecv error: recv got EOF from 127.0.0.1:45956 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260811 04:41:04.483678 27779 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.478656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.2.238.188:39937)
0811 04:41:04.478807 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.478812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.482588 (+ 3776us) server_negotiation.cc:407] Connection header received
0811 04:41:04.482949 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.482954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.483018 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.483071 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.483564 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35374: BlockingRecv error: recv got EOF from 127.0.0.1:35374 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260811 04:41:04.487728 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.486182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.2.238.189:45423)
0811 04:41:04.486291 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.486296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.486811 (+ 515us) server_negotiation.cc:407] Connection header received
0811 04:41:04.487019 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.487022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.487097 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.487152 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.487621 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45972: BlockingRecv error: recv got EOF from 127.0.0.1:45972 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260811 04:41:04.492825 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.491368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.2.238.189:45423)
0811 04:41:04.491463 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.491467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.491939 (+ 472us) server_negotiation.cc:407] Connection header received
0811 04:41:04.492112 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.492115 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.492161 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.492209 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.492708 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45976: BlockingRecv error: recv got EOF from 127.0.0.1:45976 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260811 04:41:04.498876 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.497166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45992 (local address 127.2.238.189:45423)
0811 04:41:04.497283 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.497288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.497894 (+ 606us) server_negotiation.cc:407] Connection header received
0811 04:41:04.498099 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.498102 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.498154 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.498207 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.498753 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45992: BlockingRecv error: recv got EOF from 127.0.0.1:45992 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260811 04:41:04.506721 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.505201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.2.238.189:45423)
0811 04:41:04.505358 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.505362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.505674 (+ 312us) server_negotiation.cc:407] Connection header received
0811 04:41:04.505944 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.505949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.506008 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.506059 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.506600 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45996: BlockingRecv error: recv got EOF from 127.0.0.1:45996 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260811 04:41:04.518771 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.517378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.2.238.189:45423)
0811 04:41:04.517523 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.517528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.517813 (+ 285us) server_negotiation.cc:407] Connection header received
0811 04:41:04.517966 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.517969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.518032 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.518133 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.518644 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46012: BlockingRecv error: recv got EOF from 127.0.0.1:46012 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260811 04:41:04.537951 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.536652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46026 (local address 127.2.238.189:45423)
0811 04:41:04.536802 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.536811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.536957 (+ 146us) server_negotiation.cc:407] Connection header received
0811 04:41:04.537154 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.537157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.537205 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.537285 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.537782 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46026: BlockingRecv error: recv got EOF from 127.0.0.1:46026 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260811 04:41:04.573311 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.572124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46032 (local address 127.2.238.189:45423)
0811 04:41:04.572352 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.572356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.572487 (+ 131us) server_negotiation.cc:407] Connection header received
0811 04:41:04.572598 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.572600 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.572649 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.572729 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.573184 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46032: BlockingRecv error: recv got EOF from 127.0.0.1:46032 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260811 04:41:04.640969 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.639686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.2.238.189:45423)
0811 04:41:04.639901 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.639906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.640049 (+ 143us) server_negotiation.cc:407] Connection header received
0811 04:41:04.640232 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.640237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.640294 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.640389 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.640864 (+ 475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46040: BlockingRecv error: recv got EOF from 127.0.0.1:46040 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260811 04:41:04.772619 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:04.771481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46046 (local address 127.2.238.189:45423)
0811 04:41:04.771628 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:04.771634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:04.771834 (+ 200us) server_negotiation.cc:407] Connection header received
0811 04:41:04.771902 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:04.771905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:04.771968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:04.772084 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:04.772505 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46046: BlockingRecv error: recv got EOF from 127.0.0.1:46046 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260811 04:41:05.032330 27696 negotiation.cc:337] Failed RPC negotiation. Trace:
0811 04:41:05.031085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46052 (local address 127.2.238.189:45423)
0811 04:41:05.031251 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.031255 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.031393 (+ 138us) server_negotiation.cc:407] Connection header received
0811 04:41:05.031576 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.031582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.031652 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.031734 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0811 04:41:05.032214 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46052: BlockingRecv error: recv got EOF from 127.0.0.1:46052 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260811 04:41:05.300386 27805 catalog_manager.cc:1259] Loaded cluster ID: 8ffc1788b4da4dea97d9a157635027a6
I20260811 04:41:05.300509 27805 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: loading cluster ID for follower catalog manager: success
I20260811 04:41:05.301085 27806 catalog_manager.cc:1259] Loaded cluster ID: 8ffc1788b4da4dea97d9a157635027a6
I20260811 04:41:05.301142 27806 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: loading cluster ID for follower catalog manager: success
I20260811 04:41:05.302383 27805 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: acquiring CA information for follower catalog manager: success
I20260811 04:41:05.302735 27806 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: acquiring CA information for follower catalog manager: success
I20260811 04:41:05.303633 27805 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0bf415ee0dda496caa4497eba79a170d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:41:05.303740 27806 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fae13dc630c341a78db74e8b1c5dbaed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260811 04:41:05.550567 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:05.546726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.2.238.189:45423)
0811 04:41:05.547017 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.547021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.547038 (+ 17us) server_negotiation.cc:407] Connection header received
0811 04:41:05.547189 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.547191 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.547337 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.547436 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:05.547669 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.548273 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.549089 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.549311 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.549385 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:05.549659 (+ 274us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:05.550157 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:05.550261 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:05.550309 (+ 48us) server_negotiation.cc:299] Negotiation successful
0811 04:41:05.550356 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":108,"threads_started":1}
I20260811 04:41:05.557029 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:05.553568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35384 (local address 127.2.238.188:39937)
0811 04:41:05.553890 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.553894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.553908 (+ 14us) server_negotiation.cc:407] Connection header received
0811 04:41:05.554008 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.554010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.554122 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.554221 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:05.554403 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.554909 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.555655 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.555873 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.555947 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:05.556186 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:05.556594 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:05.556714 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:05.556778 (+ 64us) server_negotiation.cc:299] Negotiation successful
0811 04:41:05.556826 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":150,"threads_started":1}
I20260811 04:41:05.563951 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:05.560085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35392 (local address 127.2.238.188:39937)
0811 04:41:05.560239 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.560244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.560544 (+ 300us) server_negotiation.cc:407] Connection header received
0811 04:41:05.560666 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.560669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.560719 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.560789 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:05.561027 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.561624 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.562649 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.563002 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.563113 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:05.563413 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:05.563504 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:05.563570 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:05.563725 (+ 155us) server_negotiation.cc:299] Negotiation successful
0811 04:41:05.563779 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260811 04:41:05.564455 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:05.560326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.2.238.189:45423)
0811 04:41:05.560456 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.560460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.560711 (+ 251us) server_negotiation.cc:407] Connection header received
0811 04:41:05.560867 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.560871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.560925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.560979 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:05.561319 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.562070 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.563134 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.563452 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.563537 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:05.563947 (+ 410us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:05.564025 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:05.564092 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:05.564246 (+ 154us) server_negotiation.cc:299] Negotiation successful
0811 04:41:05.564302 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260811 04:41:05.564764 27932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0811 04:41:05.559974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.2.238.190:44773)
0811 04:41:05.560562 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0811 04:41:05.560568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0811 04:41:05.560580 (+ 12us) server_negotiation.cc:407] Connection header received
0811 04:41:05.560671 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0811 04:41:05.560674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0811 04:41:05.560841 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0811 04:41:05.560985 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0811 04:41:05.561233 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.562041 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.563049 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0811 04:41:05.563405 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0811 04:41:05.563519 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0811 04:41:05.563984 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0811 04:41:05.564329 (+ 345us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0811 04:41:05.564416 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0811 04:41:05.564544 (+ 128us) server_negotiation.cc:299] Negotiation successful
0811 04:41:05.564607 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":74,"threads_started":1}
I20260811 04:41:05.568010 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27532
I20260811 04:41:05.576956 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27617
I20260811 04:41:05.585319 3002 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taski3hQIv/build/debug/bin/kudu with pid 27697
2026-08-11T04:41:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3198 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6304 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (283073 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST